Optimize tensor parallel execution speed by zhuohan123 · Pull Request #17 · vllm-project/vllm

zhuohan123 · 2023-03-31T15:27:00Z

Speed before this PR:

ubuntu@ray-zhuohan-cf-head-d95da8d2-compute:~/nfs/cacheflow/cacheflow$ python benchmark/benchmark_latency.py --model facebook/opt-13b
Namespace(batch_size=8, block_size=8, dtype='half', input_len=32, max_batch_size=2560, model='facebook/opt-13b', model_path='~/.cacheflow/model_weights', output_len=128, pipeline_parallel_size=1, seed=0, swap_space=20, tens
or_parallel_size=1)
2023-03-31 14:17:41,580 INFO worker.py:1535 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8266
# GPU blocks: 1975, # CPU blocks: 3276
Warm up step
Profile step: 100%|██████████████████████████████████████████████████████████████| 3/3 [00:15<00:00,  5.18s/it]
Avg latency: 5.184098243713379 seconds

Speed after this PR:

ubuntu@ray-zhuohan-cf-head-d95da8d2-compute:~/nfs/cacheflow/cacheflow$ python benchmark/benchmark_latency.py --model facebook/opt-13b
Namespace(batch_size=8, block_size=8, dtype='half', input_len=32, max_batch_size=2560, model='facebook/opt-13b', model_path='~/.cacheflow/model_weights', output_len=128, pipeline_parallel_size=1, seed=0, swap_space=20, tensor_parallel_size=1)
2023-03-31 15:20:04,885 INFO worker.py:1535 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8266
# GPU blocks: 1975, # CPU blocks: 3276
Warm up step
Profile step: 100%|██████████████████████████████████████████████████████████████| 3/3 [00:10<00:00,  3.49s/it]
Avg latency: 3.492198626200358 seconds

WoosukKwon

Awesome! Thanks for the effort.

Rebase fp8_kv branch with upstream (3-07-2024)

These Dockerfile changes: - Update the release stage to work with the recently refactored `requirements-common.txt` / `requirements-cuda.txt` split - Fixup the kernel compilation in the `build` stage to correctly pick up cuda - Install the kernels from this docker build rather than pulling a precompiled wheel. We can swap that back once a new wheel is available with the correct pytorch version + updated interfaces --------- Signed-off-by: Nick Hill <nickhill@us.ibm.com> Signed-off-by: Joe Runde <Joseph.Runde@ibm.com> Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>

[ROCm] adding a missing triton autotune config

### What this PR does / why we need it? Add dispatch key for NPU, so that the log could be print correctly. Now ``` executor_base.py:110] # CPU blocks: 220478, # CPU blocks: 21845 ``` After this pr ``` executor_base.py:110] # NPU blocks: 220478, # CPU blocks: 21845 ``` ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? CI passed and log printed as above Signed-off-by: MengqingCao <cmq0113@163.com>

Load balance across multiple workers

* prefill mla Signed-off-by: Chen Zhang <zhangch99@outlook.com> * can run now Signed-off-by: Chen Zhang <zhangch99@outlook.com> * tmp Signed-off-by: Chen Zhang <zhangch99@outlook.com> * can output the first token Signed-off-by: Chen Zhang <zhangch99@outlook.com> * fix bug Signed-off-by: Chen Zhang <zhangch99@outlook.com> * remove some debug Signed-off-by: Chen Zhang <zhangch99@outlook.com> * update Signed-off-by: Chen Zhang <zhangch99@outlook.com> * hack through cu_seqlen_ks exploding issue * update basic.py Signed-off-by: Chen Zhang <zhangch99@outlook.com> * remove some unnecessary changes Signed-off-by: Chen Zhang <zhangch99@outlook.com> * clean up Signed-off-by: Chen Zhang <zhangch99@outlook.com> --------- Signed-off-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>

401 of the step's 402 cases need no device, but test_scheduler.py::test_async_scheduling_pp_allows_rescheduling_with_output_placeholders builds a ParallelConfig with pipeline_parallel_size=2. It never touches a second card - the step allocates 0.0 GiB across its whole run - but on a CUDA image ParallelConfig validates world size against the visible GPU count and rejects 2. Upstream never sees this: the step depends on image-build-cpu, where that check is inert. So the marking upstream gives a step describes the image it runs on, not what its cases ask for, and _gpus_for() had no way to say otherwise. Adds docker.devices_by_step, a per-key override of the device count that indexes into the existing gpus_by_devices map. Only v1-others-cpu uses it; the generated pipeline differs from before by exactly one line, and the other CPU-marked steps still get "device=0". Two cards costs nothing when the step allocates no VRAM, and leaving it at one meant a guaranteed failure every run - which is expensive now that fail_fast is back on. In vllm-project#24 it cancelled Basic Models Test and Torch Stable ABI Audit, both of which pass (3m19s in vllm-project#17). I had written this case off as "not worth handling" without accounting for that. Co-authored-by: Claude Opus 5

Adds TritonNvFp4LinearKernel: dequantizes packed e2m1 weights (2 codes per uint8 byte, low-nibble-first, per-16-group e4m3 block scale, scalar global scale) directly in the GEMM inner loop and consumes bf16/fp16 activations unquantized -- the W4A16 weight-only path used by nvfp4-pack-quantized compressed-tensors checkpoints (e.g. huginnfork/Qwen3.8-27B-NVFP4A16, whose config.json has input_activations: null) and ModelOpt's W4A16_NVFP4 quant_method. The decode math is ported from the standalone prototype/benchmark that measured it (nvfp4_final.py, fp8-decode's vllm-project#19 pass: correct on a synthetic K=512,N=256 sweep, ~3.5 decode ops/weight, 2.26x wall-clock over an Emulation-style dequant+matmul baseline on gate_up-shaped data), not re-derived. Two things needed changing to productize it and are NOT yet GPU-verified in this form, flagged inline in the new file: - Packing layout: the prototype's synthetic test data packs 8 e2m1 codes per int32. Real checkpoints pack 2 codes per uint8 byte along K, confirmed against torch.float4_e2m1fn_x2's own unpack convention. The unpack is rewritten for byte-pair packing; the bit-shift E2M1 decode, the fp16-subnormal-flush defense (gfx90a's MFMA flushes subnormal inputs to zero, same finding as the FP8 W8A16 kernel, 2**14 fold here vs that kernel's 2**8), and the two-stage scale fold are unchanged from the validated prototype. - weight_global_scale: the prototype's harness had no global scale (its synthetic data didn't model one). Real NVFP4 checkpoints do; added as a third fp32 epilogue multiply, not present in the validated prototype. Also fixes the dispatcher landmine that blocked ANY weight-only NVFP4 on ROCm regardless of what's registered in _POSSIBLE_NVFP4_KERNELS: init_nvfp4_linear_kernel's linear_backend=="auto" and use_a16 branch force-selected MarlinNvFp4LinearKernel unconditionally, with no platform check, and raised (Marlin is CUDA-only) before the function ever reached the registry search below it. Same class of bug as d74ce3d's CUDA-shaped capability check, just in the NVFP4 dispatcher instead of the FP8 scheme-selection path. Gated the force on current_platform.is_cuda() so other platforms fall through to their own registry entry. TritonNvFp4LinearKernel is added to both _POSSIBLE_NVFP4_KERNELS[ROCM] (ahead of EmulationNvFp4LinearKernel) and the separate a16_kernels tuple in init_nvfp4_linear_kernel -- registering in only one silently drops the kernel from use_a16=True requests via the other's filter. Gated strictly to gfx90a via on_gfx90a(), same as TritonW8A16Fp8LinearKernel: the subnormal-flush workaround and any future tile-ladder tuning are measured facts about this card, not proven elsewhere. NOT YET RUN ON GPU. Per team sequencing the card is currently occupied by the FP8 pipeline (e2e battery, vllm-project#17 benchmark, fp8-impl's ladder measurements); this lands as a reviewable, buildable artifact for its own verification pass once cleared, by someone other than its author. Does not ship in the FP8 PRs -- separate verification cycle, separate PR. MXFP4 is the natural follow-on (E8M0 per-32 scales instead of e4m3 per-16, cheaper decode -- E8M0 dequant is a pure exponent add, no mantissa/sign bits at all -- gpt-oss checkpoint format) but is intentionally not implemented here; ROCm already has a non-emulation MXFP4 backend via AiterMxfp4LinearKernel, gated off on this card only by supports_mx() excluding gfx90a, and MXFP4's own init_mxfp4_linear_kernel has no analogous CUDA-hardcoded force to fix -- a smaller, separate task.

ci: build vLLM on a self-hosted CN runner

…ive reader Counts K/V elements that would saturate E4M3 after scaling. The increments are pure tensor ops inside the forward, safe under capture by construction. Reading happens only where concurrency cannot exist: VLLM_QSA_KV_CLIP_COUNT arms the counters and drains per-layer totals once at engine shutdown on the engine's own thread; VLLM_QSA_KV_CLIP_READER additionally runs the live reader thread, diagnostic-only under --enforce-eager. The reader constraint is architectural, established across four engine-killing failure modes, each the next CUDA rule a concurrent thread violates during graph capture: sync reads (halt95 issue vllm-project#17), pinned-host allocation, stream creation draining the device, and event queries — all illegal or hazardous during capture, and no API exposes capture state to another thread. halt95's rc3-rc7 patch arc (non-blocking mirrors, 0071) reached the same conclusion and their 0073 operator ruling shipped readers off; this design ports that ruling. Tests drive _qsa_clip_count as production does, arming under inference_mode and asserting the drained world. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Adrian Zankich <adzankich@gmail.com>

…e outbound transfer per card per round Replaces the grouped all-to-all (each card receiving from all three peers at once), which every V620 bus drop since 09-25 had enabled. Bit-identical results; 1.02-1.15x faster on 4x V620 (q8_ar_test.py). VLLM_RDNA_AR_Q8_STAGGER=0 restores the old exchange. CHANGES vllm-project#17. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ive reader Counts K/V elements that would saturate E4M3 after scaling. The increments are pure tensor ops inside the forward, safe under capture by construction. Reading happens only where concurrency cannot exist: VLLM_QSA_KV_CLIP_COUNT arms the counters and drains per-layer totals once at engine shutdown on the engine's own thread; VLLM_QSA_KV_CLIP_READER additionally runs the live reader thread, diagnostic-only under --enforce-eager. The reader constraint is architectural, established across four engine-killing failure modes, each the next CUDA rule a concurrent thread violates during graph capture: sync reads (halt95 issue vllm-project#17), pinned-host allocation, stream creation draining the device, and event queries — all illegal or hazardous during capture, and no API exposes capture state to another thread. halt95's rc3-rc7 patch arc (non-blocking mirrors, 0071) reached the same conclusion and their 0073 operator ruling shipped readers off; this design ports that ruling. Tests drive _qsa_clip_count as production does, arming under inference_mode and asserting the drained world. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Adrian Zankich <adzankich@gmail.com>

[Step5] Require Model Runner V2 and drop the V1 runner changes

…plane [Feature][Lwd] Build the edge-cloud parallel groups from the LWD topology without two-stage PP

vllm-project#17) Keep fixed-ring KPool tail state out of context-length token slot mapping. Retain dedicated tail writes and add boundary regression tooling for chunked prefill. Preserve the reviewed source tree from PR vllm-project#17; the squash only reorganizes Git history. Existing acceptance evidence and remaining limitations are recorded in overview PR vllm-project#11. No new model tests were run for this integration. Original-PR-Head: cc3ea839105a676b1071e377b3cd61c0427c2d47 Assisted-by: Codex

The grouped-GEMM permute (ep_scatter) launched one program per token and let every thread of the block redo the same 8 scalar top-k lookups, so a layer paid ~375 us per permute at 8k tokens while only ~30 us of that was actual row movement. ep_gather had the same shape problem (one program per hidden chunk and token block, with a token loop inside). These kernels are the MUSA implementation of the permute/gather, selected through current_platform.is_musa() like the other MUSA paths in this fork: * route pairs with a 64-token tile so the top-k loads coalesce and the destination atomics are issued as vectors; * compute the per-expert start offsets on the device instead of with torch ops; * copy the activation row together with its block scales in one kernel, driven by a compact source-row map, so the scatter is a plain vectorised row copy; * gather with one program per (token, hidden chunk) and a static top-k loop. Measured on 8x MTT S5000, GLM-5.3-Flash, TP1 + PCP8 + EP8, 128 prompts x 8192 in: permute 375 -> 120 us (3.1x), gather 128 -> 119 us (1.07x, bitwise identical) end to end 16,744 -> 17,302 tok/s (+3.3%), median TTFT 30.99 -> 29.98 s Correctness: every routed pair was compared against the stock kernel (activation row, scales and m_indices all match) and greedy generation is token-identical to the baseline (5/5 prompts, max |d mean logprob| = 5.35e-3). Other platforms keep the existing kernels unchanged.

vllm-project#17) Keep fixed-ring KPool tail state out of context-length token slot mapping. Retain dedicated tail writes and add boundary regression tooling for chunked prefill. Preserve the reviewed source tree from PR vllm-project#17; the squash only reorganizes Git history. Existing acceptance evidence and remaining limitations are recorded in overview PR vllm-project#11. No new model tests were run for this integration. Original-PR-Head: cc3ea839105a676b1071e377b3cd61c0427c2d47 Assisted-by: Codex

The grouped-GEMM permute (ep_scatter) launched one program per token and let every thread of the block redo the same 8 scalar top-k lookups, so a layer paid ~375 us per permute at 8k tokens while only ~30 us of that was actual row movement. ep_gather had the same shape problem (one program per hidden chunk and token block, with a token loop inside). These kernels are the MUSA implementation of the permute/gather, selected through current_platform.is_musa() like the other MUSA paths in this fork: * route pairs with a 64-token tile so the top-k loads coalesce and the destination atomics are issued as vectors; * compute the per-expert start offsets on the device instead of with torch ops; * copy the activation row together with its block scales in one kernel, driven by a compact source-row map, so the scatter is a plain vectorised row copy; * gather with one program per (token, hidden chunk) and a static top-k loop. Measured on 8x MTT S5000, GLM-5.3-Flash, TP1 + PCP8 + EP8, 128 prompts x 8192 in: permute 375 -> 120 us (3.1x), gather 128 -> 119 us (1.07x, bitwise identical) end to end 16,744 -> 17,302 tok/s (+3.3%), median TTFT 30.99 -> 29.98 s Correctness: every routed pair was compared against the stock kernel (activation row, scales and m_indices all match) and greedy generation is token-identical to the baseline (5/5 prompts, max |d mean logprob| = 5.35e-3). Other platforms keep the existing kernels unchanged.

vllm-project#17) Keep fixed-ring KPool tail state out of context-length token slot mapping. Retain dedicated tail writes and add boundary regression tooling for chunked prefill. Preserve the reviewed source tree from PR vllm-project#17; the squash only reorganizes Git history. Existing acceptance evidence and remaining limitations are recorded in overview PR vllm-project#11. No new model tests were run for this integration. Original-PR-Head: cc3ea839105a676b1071e377b3cd61c0427c2d47 Assisted-by: Codex

The grouped-GEMM permute (ep_scatter) launched one program per token and let every thread of the block redo the same 8 scalar top-k lookups, so a layer paid ~375 us per permute at 8k tokens while only ~30 us of that was actual row movement. ep_gather had the same shape problem (one program per hidden chunk and token block, with a token loop inside). These kernels are the MUSA implementation of the permute/gather, selected through current_platform.is_musa() like the other MUSA paths in this fork: * route pairs with a 64-token tile so the top-k loads coalesce and the destination atomics are issued as vectors; * compute the per-expert start offsets on the device instead of with torch ops; * copy the activation row together with its block scales in one kernel, driven by a compact source-row map, so the scatter is a plain vectorised row copy; * gather with one program per (token, hidden chunk) and a static top-k loop. Measured on 8x MTT S5000, GLM-5.3-Flash, TP1 + PCP8 + EP8, 128 prompts x 8192 in: permute 375 -> 120 us (3.1x), gather 128 -> 119 us (1.07x, bitwise identical) end to end 16,744 -> 17,302 tok/s (+3.3%), median TTFT 30.99 -> 29.98 s Correctness: every routed pair was compared against the stock kernel (activation row, scales and m_indices all match) and greedy generation is token-identical to the baseline (5/5 prompts, max |d mean logprob| = 5.35e-3). Other platforms keep the existing kernels unchanged.

…lm-project#17) The DSA indexer is not torch.compile'd, so the torch.ops.vllm.index_rope registration and the is_compiling branch are unused. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

zhuohan123 added 2 commits March 31, 2023 15:25

Optimize tensor parallel execution speed

a32f244

add more files

c3e6bce

zhuohan123 requested a review from WoosukKwon March 31, 2023 15:32

WoosukKwon approved these changes Mar 31, 2023

View reviewed changes

WoosukKwon reviewed Mar 31, 2023

View reviewed changes

Comment thread benchmark/benchmark_latency.py Resolved

nit

2bea93e

zhuohan123 merged commit c45f3c3 into main Mar 31, 2023

zhuohan123 deleted the optimize-tp-speed branch June 18, 2023 07:22

shanshanpt mentioned this pull request Nov 17, 2023

Run long conetxt error : CUDA error: an illegal memory access was encountered #1700

Closed

junior-zsy mentioned this pull request Nov 20, 2023

Error with 32k Long Text in chatglm2-6b-32k Model #1725

Closed

hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024

Optimize tensor parallel execution speed (vllm-project#17)

ad3d36f

AdrianAbeyta referenced this pull request in ROCm/vllm Mar 8, 2024

Merge pull request #17 from ROCm/IFU-2024-03-01-fp8-kv

b3d81e0

Rebase fp8_kv branch with upstream (3-07-2024)

fxmarty pushed a commit to fxmarty/vllm-public that referenced this pull request May 31, 2024

Merge pull request vllm-project#17 from ROCm/triton-config-fix

bebcbe6

[ROCm] adding a missing triton autotune config

alixiaodi mentioned this pull request Aug 2, 2024

[Bug]: #7072

Closed

SpaceHunterInf mentioned this pull request Sep 30, 2024

[Bug]: Bus error (core dumped) #8974

Closed

1 task done

hao-cold mentioned this pull request May 13, 2025

[Bug]: CUDA error: an illegal instruction was encountered #18045

Closed

1 task done

markmc mentioned this pull request May 21, 2025

[Bug][Failing Test]: Distributed Comm Ops - distributed/test_shm_broadcast.py #18492

Closed

1 task done

zerosurplus mentioned this pull request Jun 16, 2025

[Bug]: torch.distributed.DistNetworkError: The client socket has timed out after 600000ms while trying to connect to (172.17.0.9, 46229). #19670

Closed

1 task done

robertgshaw2-redhat added a commit that referenced this pull request Jul 7, 2025

Merge pull request #17 from praveingk/batching

39e6bd1

Load balance across multiple workers

xiaomofang mentioned this pull request Jul 31, 2025

[Bug]: There is an issue with speculative inference in Eagle mode, where the context length of vLLM inference is constrained by the draft model. #21986

Closed

1 task done

zyongye pushed a commit to zyongye/vllm that referenced this pull request Aug 5, 2025

Add TRT-LLM Attention Sink and MXFP4 MoE (vllm-project#17)

78e69f6

zyongye pushed a commit to zyongye/vllm that referenced this pull request Aug 6, 2025

Add TRT-LLM Attention Sink and MXFP4 MoE (vllm-project#17)

2cc41a7

JeffreyWong20 mentioned this pull request Aug 19, 2025

[Bug]: [TPU] profiling_tpu/profiling.py example crashed when runs on vllm_tpu docker #23194

Closed

1 task done

Michel-debug mentioned this pull request Oct 23, 2025

[Bug]: qwen3-vl-2b after ms-swift fine-tuning lance errors #27405

Closed

1 task done

acodercat mentioned this pull request Nov 10, 2025

[Bugfix] Add strong reference to CUDA pluggable allocator callbacks #23477

Merged

4 tasks

davetha mentioned this pull request Aug 15, 2026

[ROCm] Fused Triton W8A16 fp8 GEMM for gfx90a + CDNA2 fp8 checkpoint fallback davetha/vllm#7

Merged

grYe99 pushed a commit to grYe99/vllm that referenced this pull request Sep 17, 2026

Merge pull request vllm-project#17 from DaoCloud/ci/selfhosted-build

56fd0b7

ci: build vLLM on a self-hosted CN runner

xzwgit mentioned this pull request Sep 30, 2026

[Bugfix][Kernel] Bump DeepGEMM pin for SM120 32-state page support (DeepGEMM#14) #59385

Closed

lisawym mentioned this pull request Oct 1, 2026

[Bugfix][Kernel] Bump DeepGEMM pin for SM107 JIT compatibility #59503

Open

5 of 6 tasks

YukioZzz pushed a commit to YukioZzz/vllm that referenced this pull request Oct 16, 2026

Merge pull request vllm-project#17 from jeejeelee/step5-mrv2-only

626ae87

[Step5] Require Model Runner V2 and drop the V1 runner changes

tjtanaa mentioned this pull request Oct 26, 2026

[RFC]: Attention Backend Test Suites #63640

Open

1 task done

Chloe-orz pushed a commit to Chloe-orz/vllm-maintomain that referenced this pull request Oct 26, 2026

Merge pull request vllm-project#17 from vllm-data-comm/feat/lwd-data-…

4ee8aec

…plane [Feature][Lwd] Build the edge-cloud parallel groups from the LWD topology without two-stage PP

abmfy mentioned this pull request Oct 28, 2026

[Perf][DSv4.1] Improve the TP4 concurrency-1 specialized decode #64364

Open

2 of 6 tasks

Navigation Menu

Repository navigation

Optimize tensor parallel execution speed - #17

Optimize tensor parallel execution speed#17
zhuohan123 merged 3 commits into
mainfrom
optimize-tp-speed

zhuohan123 commented Mar 31, 2023

Uh oh!

WoosukKwon left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zhuohan123 commented Mar 31, 2023

Uh oh!

WoosukKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants