Skip to content

[Elastic EP] Async preparation - #47288

Merged
vllm-bot merged 17 commits into
vllm-project:mainfrom
itayalroy:async_eep
Jul 28, 2026
Merged

vllm-bot merged 17 commits into
vllm-project:mainfrom
itayalroy:async_eep

Conversation

@itayalroy

@itayalroy itayalroy commented Jul 1, 2026 •

Copy link
Copy Markdown
Contributor

Async Elastic EP preparation

Summary

This PR moves Elastic EP preparation off the main serving thread. Creating
standby process groups, setting up EP communication, staging quant methods,
transferring model weights, synchronizing KV-cache memory, and preparing new
workers now happen on background worker threads. Requests continue to be served
regularly during this work and are blocked briefly only for the final switch.

This also makes the implementation much simpler. Preparation steps are no longer
coordinated with forward passes, so staged barriers, notification flows and
cross dp synchronization are no longer needed. The result is a simpler two-step
flow: background preparation followed by a synchronous commit.

This PR also removes coord_store_port from the AOT hash calculation, as this
port changes on every run and prevents reuse of cached Torch compilation artifacts.

Performance

Single-node scale-up using NIXL EP, DeepSeek-V2-Lite-Chat on NVIDIA H100
80 GB GPUs, from DP 4 to DP 8.

Light traffic uses one client with one request in flight at a time. Heavy
traffic uses one such client per DP rank. CUDA graphs test assumes
pre-warmed torch compilation cache (automatically cached in the first
run on the machine).

Total time is the full reconfiguration time. Downtime is measured by polling
/is_scaling_elastic_ep, from the first 503 until the first 200 after the
last 503.

Test Before: total time This PR: total time Before: downtime This PR: downtime Downtime reduction
Eager mode, no traffic 35.653 s 37.338 s 35.653 s 7.324 s 79%
Eager mode, light traffic 38.036 s 36.997 s 38.036 s 6.600 s 83%
Eager mode, heavy traffic 38.089 s 38.042 s 38.089 s 6.584 s 83%
CUDA graphs, heavy traffic* 50.183 s 42.069 s 50.183 s 12.556 s 75%

* Assumes a pre-warmed Torch compilation cache, which is automatically populated during the first run on the machine.

Validation

The Elastic EP test was extended to cover eager mode with no, light, and heavy
traffic, plus CUDA graphs under heavy traffic. It checks GSM8K accuracy before
scaling and after scale-up and scale-down, and reports downtime and total
reconfiguration time.

Internally, we also validate allgather_reducescatter and nixl_ep
across different traffic patterns, including additional tests such as repeated scaling
with accuracy checks after every resize- these tests can be upstreamed as well in
follow-up PRs but are probably too long for vLLM CI.

Follow-ups

  • Move the post-reconfiguration EPLB work to async execution.
    This further reduces the remaining 6-7 seconds of downtime. It is
    deferred to a follow-up PR to keep this PR reviewable.
  • Reduce downtime without --enforce-eager. In single-node tests, recapturing
    CUDA graphs takes ~6 seconds, while torch compilation adds additional
    ~10-20 seconds. Pre-warming the torch compile cache can avoid most of the
    compilation time, but CUDA graph recapture and kernel warmups still block traffic.
  • Further benchmark and optimize multi-node EEP reconfiguration downtime

cc @tlrmchlsmth @SageMoore @ilmarkov

@itayalroy
itayalroy requested a review from njhill as a code owner July 1, 2026 11:13

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added frontend nvidia intel-gpu Related to Intel GPU cpu Related to CPU backends labels Jul 1, 2026
@mergify mergify Bot added the v1 label Jul 1, 2026
Comment thread vllm/v1/engine/async_llm.py Resolved
@tlrmchlsmth tlrmchlsmth self-assigned this Jul 1, 2026
@tlrmchlsmth

Copy link
Copy Markdown
Member

FYI @SageMoore @ilmarkov

@itayalroy
itayalroy force-pushed the async_eep branch 2 times, most recently from 62eb3c2 to 86ce796 Compare July 2, 2026 07:58
Comment thread vllm/v1/engine/core_client.py Resolved
@mergify

mergify Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @itayalroy.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Comment thread vllm/v1/engine/async_llm.py Resolved
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
auto-merge was automatically disabled July 27, 2026 22:15

Head branch was pushed to by a user without write access

@tlrmchlsmth
tlrmchlsmth enabled auto-merge (squash) July 27, 2026 23:58
@mergify mergify Bot removed the needs-rebase label Jul 28, 2026
@vllm-bot
vllm-bot merged commit 35efdf6 into vllm-project:main Jul 28, 2026
125 of 128 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Jul 28, 2026
1542305589 added a commit to 1542305589/vllm that referenced this pull request Jul 30, 2026
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
linfeng-yuan pushed a commit to vllm-project/vllm-ascend that referenced this pull request Aug 3, 2026
### What this PR does / why we need it?

| Files | Upstream vLLM change | vllm-ascend adaptation |
|-------|---------------------|------------------------|
| `vllm_ascend/__init__.py` |
[vllm#48841](vllm-project/vllm#48841) — added
`from triton.experimental import gluon`, `from triton.experimental.gluon
import language as gl` and `from triton.language.core import _aggregate`
to `triton_utils/__init__.py`, requiring Triton 3.6+ API absent from
`triton-ascend 3.2.1` | Pre-register `triton.experimental.gluon` /
`.language` as `sys.modules` stubs; stub
`triton.language.core._aggregate`. Gated on `os.getenv("VLLM_VERSION")
!= "0.26.0"`, mirroring `vllm_version_is` env var path. |
| `tests/ut/patch/platform/test_patch_structured_output.py` |
[vllm#49665](vllm-project/vllm#49665) — changed
`VLLMValidationError` base class from `ValueError` to `VLLMClientError`,
breaking `pytest.raises(ValueError)` assertions | Updated 3 assertions
from `pytest.raises(ValueError, ...)` to
`pytest.raises(VLLMValidationError, ...)` |
| `tests/e2e/pull_request/one_card/test_guided_decoding.py` |
[vllm#49665](vllm-project/vllm#49665) — same
`VLLMValidationError` base-class change; on 0.26.0 the upstream
validation may still raise `ValueError` | Version-gated assertion:
`pytest.raises(ValueError, ...)` on 0.26.0 vs
`pytest.raises(VLLMValidationError, ...)` on main |
|
`vllm_ascend/ops/vocab_parallel_embedding.py`<br>`vllm_ascend/_310p/ops/vocab_parallel_embedding.py`
| [vllm#49731](vllm-project/vllm#49731) — added
`*, disable_tp: bool = False` keyword arg to `ParallelLMHead.__init__()`
and `VocabParallelEmbedding.__init__()` | Version-gated `__init__` with
`vllm_version_is("0.26.0")`: else branch accepts `disable_tp` kwarg and
forwards it (Ascend manages TP via `lmhead_tp_enable()`) |
|
`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py`<br>`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_hybrid_connector.py`
| [vllm#26593](vllm-project/vllm#26593) —
`KVConnectorBase_V1.__init__()` sets `self._kv_transfer_config`,
required by `requires_kv_delivery` property | Version-gated `__init__`:
else branch sets `self._kv_transfer_config` on vllm main |
| `vllm_ascend/ops/fused_moe/fused_moe.py` |
[vllm#50089](vllm-project/vllm#50089) — added
`fused_output_is_reduced` kwarg to
`_maybe_reduce_shared_expert_output()` and `output_is_reduced` kwarg to
`_maybe_reduce_final_output()` | Version-gated both methods: else branch
accepts new kwargs but ignores them (Ascend handles reduction
independently); `trunc_size` may now be `None` |
| `vllm_ascend/patch/worker/patch_distributed.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `GroupCoordinator.__init__()` |
Version-gated via `_IS_VLLM_026` env var check (avoids vllm-ascend
import cycle). Else branch stores `self.use_all2all` (unused, no all2all
on Ascend) |
| `vllm_ascend/distributed/device_communicators/npu_communicator.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `DeviceCommunicatorBase.__init__()`
| Version-gated `NPUCommunicator.__init__` with
`vllm_version_is("0.26.0")`: else branch accepts and forwards
`use_all2all` (NPU keeps the no-op `_NpuAll2AllManager`) |
| `vllm_ascend/worker/npu_input_batch.py` |
[vllm#48018](vllm-project/vllm#48018) — added
`use_replayssm: bool` kwarg (Mamba replay-SSM);
[vllm#40996](vllm-project/vllm#40996) — added
`slot_mapping_modes` kwarg (DCP hybrid attention) | Accepts both kwargs
unconditionally; stores `self.use_replayssm` and
`self.slot_mapping_modes` only on main (`not vllm_version_is("0.26.0")`)
for interface alignment |
| `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` |
[vllm#49364](vllm-project/vllm#49364) — renamed
`skip_attn` parameter to `full_cudagraph` in cudagraph mode logic |
Version-gated call: 0.26.0 keeps `skip_attn=(cg_mode != PIECEWISE)`;
main passes `full_cudagraph=(cg_mode != PIECEWISE)` |
|
`vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py`<br>`vllm_ascend/distributed/weight_transfer/__init__.py`
| [vllm#48042](vllm-project/vllm#48042) +
[vllm#48981](vllm-project/vllm#48981) — replaced
static `IPCTrainerSendWeightsArgs`/`trainer_send_weights` with a
stateful `IPCTrainerWeightTransferEngine` driven by
`WeightTransferTrainerFactory.trainer_init(...).send_weights()`;
`packed` moved onto trainer init-info; per-round `update_info` slimmed |
Whole module version-gated: 0.26.0 keeps the static
`NPUIPCWeightTransferEngine`; main defines stateful
`NPUIPCTrainerWeightTransferEngine` (`_send_unpacked` instance method,
`packed` on `NPUIPCWeightTransferInitInfo`, no-arg
`npu_generate_uuid()`, `is_sender`/`_all_gather_and_merge_handles`).
`register_engine()` also registers the trainer engine in
`WeightTransferTrainerFactory` on main only |
| `examples/rl/rlhf_http_npu_ipc.py` |
[vllm#48981](vllm-project/vllm#48981) — ported
RL examples to `WeightTransferTrainerFactory.trainer_init(...)` +
`engine.send_weights()` with `HTTPVLLMWeightSyncClient`/`ModuleSource` |
Version-gated example: 0.26.0 uses static
`NPUIPCWeightTransferEngine.trainer_send_weights`; main uses the
stateful trainer engine path |
| `tests/e2e/pull_request/one_card/test_npu_ipc_weight_transfer.py` |
[vllm#48981](vllm-project/vllm#48981) — IPC
weight-transfer E2E now exercised via the stateful trainer engine |
Version-gated test: 0.26.0 keeps the manual `_post(start/finish)` +
static `trainer_send_weights`; main drives
`WeightTransferTrainerFactory.trainer_init(...).send_weights()` (engine
owns the lifecycle) |
| `tests/ut/distributed/weight_transfer/test_npu_ipc_engine.py` |
[vllm#48981](vllm-project/vllm#48981) —
trainer-side IPC APIs changed from static methods to instance methods on
the stateful engine | Version-gated tests (`IS_VLLM_026`):
`_send_unpacked`, `packed` placement, `npu_generate_uuid` arity, and
`start/finish_weight_update` no-op behavior asserted per version |
| `vllm_ascend/worker/v2/spec_decode/dflash/speculator.py` |
[vllm#50000](vllm-project/vllm#50000) —
`_prepare_dflash_inputs_kernel` gained `temperature`/`seeds` sampling
params for probabilistic draft sampling | Version-gated
`_prepare_dflash_inputs_kernel_ascend`: 0.26.0 branch keeps the old
signature; main branch adds the four `temperature`/`seeds` pointers and
the corresponding stores to stay aligned with upstream while keeping
Ascend's own kernel |
| `.github/vllm-main-verified.commit` | — | Updated verified main commit
hash to `0351e9aa1fdf1a51329d1906881528dfe61fc88e` |

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?


- vLLM version: v0.26.0
- vLLM main:
vllm-project/vllm@d02df74

---------

Signed-off-by: hfadzxy <starmoon_zhang@163.com>
HMCCMH pushed a commit to hotTea123/vllm-ascend that referenced this pull request Aug 12, 2026
### What this PR does / why we need it?

| Files | Upstream vLLM change | vllm-ascend adaptation |
|-------|---------------------|------------------------|
| `vllm_ascend/__init__.py` |
[vllm#48841](vllm-project/vllm#48841) — added
`from triton.experimental import gluon`, `from triton.experimental.gluon
import language as gl` and `from triton.language.core import _aggregate`
to `triton_utils/__init__.py`, requiring Triton 3.6+ API absent from
`triton-ascend 3.2.1` | Pre-register `triton.experimental.gluon` /
`.language` as `sys.modules` stubs; stub
`triton.language.core._aggregate`. Gated on `os.getenv("VLLM_VERSION")
!= "0.26.0"`, mirroring `vllm_version_is` env var path. |
| `tests/ut/patch/platform/test_patch_structured_output.py` |
[vllm#49665](vllm-project/vllm#49665) — changed
`VLLMValidationError` base class from `ValueError` to `VLLMClientError`,
breaking `pytest.raises(ValueError)` assertions | Updated 3 assertions
from `pytest.raises(ValueError, ...)` to
`pytest.raises(VLLMValidationError, ...)` |
| `tests/e2e/pull_request/one_card/test_guided_decoding.py` |
[vllm#49665](vllm-project/vllm#49665) — same
`VLLMValidationError` base-class change; on 0.26.0 the upstream
validation may still raise `ValueError` | Version-gated assertion:
`pytest.raises(ValueError, ...)` on 0.26.0 vs
`pytest.raises(VLLMValidationError, ...)` on main |
|
`vllm_ascend/ops/vocab_parallel_embedding.py`<br>`vllm_ascend/_310p/ops/vocab_parallel_embedding.py`
| [vllm#49731](vllm-project/vllm#49731) — added
`*, disable_tp: bool = False` keyword arg to `ParallelLMHead.__init__()`
and `VocabParallelEmbedding.__init__()` | Version-gated `__init__` with
`vllm_version_is("0.26.0")`: else branch accepts `disable_tp` kwarg and
forwards it (Ascend manages TP via `lmhead_tp_enable()`) |
|
`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py`<br>`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_hybrid_connector.py`
| [vllm#26593](vllm-project/vllm#26593) —
`KVConnectorBase_V1.__init__()` sets `self._kv_transfer_config`,
required by `requires_kv_delivery` property | Version-gated `__init__`:
else branch sets `self._kv_transfer_config` on vllm main |
| `vllm_ascend/ops/fused_moe/fused_moe.py` |
[vllm#50089](vllm-project/vllm#50089) — added
`fused_output_is_reduced` kwarg to
`_maybe_reduce_shared_expert_output()` and `output_is_reduced` kwarg to
`_maybe_reduce_final_output()` | Version-gated both methods: else branch
accepts new kwargs but ignores them (Ascend handles reduction
independently); `trunc_size` may now be `None` |
| `vllm_ascend/patch/worker/patch_distributed.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `GroupCoordinator.__init__()` |
Version-gated via `_IS_VLLM_026` env var check (avoids vllm-ascend
import cycle). Else branch stores `self.use_all2all` (unused, no all2all
on Ascend) |
| `vllm_ascend/distributed/device_communicators/npu_communicator.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `DeviceCommunicatorBase.__init__()`
| Version-gated `NPUCommunicator.__init__` with
`vllm_version_is("0.26.0")`: else branch accepts and forwards
`use_all2all` (NPU keeps the no-op `_NpuAll2AllManager`) |
| `vllm_ascend/worker/npu_input_batch.py` |
[vllm#48018](vllm-project/vllm#48018) — added
`use_replayssm: bool` kwarg (Mamba replay-SSM);
[vllm#40996](vllm-project/vllm#40996) — added
`slot_mapping_modes` kwarg (DCP hybrid attention) | Accepts both kwargs
unconditionally; stores `self.use_replayssm` and
`self.slot_mapping_modes` only on main (`not vllm_version_is("0.26.0")`)
for interface alignment |
| `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` |
[vllm#49364](vllm-project/vllm#49364) — renamed
`skip_attn` parameter to `full_cudagraph` in cudagraph mode logic |
Version-gated call: 0.26.0 keeps `skip_attn=(cg_mode != PIECEWISE)`;
main passes `full_cudagraph=(cg_mode != PIECEWISE)` |
|
`vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py`<br>`vllm_ascend/distributed/weight_transfer/__init__.py`
| [vllm#48042](vllm-project/vllm#48042) +
[vllm#48981](vllm-project/vllm#48981) — replaced
static `IPCTrainerSendWeightsArgs`/`trainer_send_weights` with a
stateful `IPCTrainerWeightTransferEngine` driven by
`WeightTransferTrainerFactory.trainer_init(...).send_weights()`;
`packed` moved onto trainer init-info; per-round `update_info` slimmed |
Whole module version-gated: 0.26.0 keeps the static
`NPUIPCWeightTransferEngine`; main defines stateful
`NPUIPCTrainerWeightTransferEngine` (`_send_unpacked` instance method,
`packed` on `NPUIPCWeightTransferInitInfo`, no-arg
`npu_generate_uuid()`, `is_sender`/`_all_gather_and_merge_handles`).
`register_engine()` also registers the trainer engine in
`WeightTransferTrainerFactory` on main only |
| `examples/rl/rlhf_http_npu_ipc.py` |
[vllm#48981](vllm-project/vllm#48981) — ported
RL examples to `WeightTransferTrainerFactory.trainer_init(...)` +
`engine.send_weights()` with `HTTPVLLMWeightSyncClient`/`ModuleSource` |
Version-gated example: 0.26.0 uses static
`NPUIPCWeightTransferEngine.trainer_send_weights`; main uses the
stateful trainer engine path |
| `tests/e2e/pull_request/one_card/test_npu_ipc_weight_transfer.py` |
[vllm#48981](vllm-project/vllm#48981) — IPC
weight-transfer E2E now exercised via the stateful trainer engine |
Version-gated test: 0.26.0 keeps the manual `_post(start/finish)` +
static `trainer_send_weights`; main drives
`WeightTransferTrainerFactory.trainer_init(...).send_weights()` (engine
owns the lifecycle) |
| `tests/ut/distributed/weight_transfer/test_npu_ipc_engine.py` |
[vllm#48981](vllm-project/vllm#48981) —
trainer-side IPC APIs changed from static methods to instance methods on
the stateful engine | Version-gated tests (`IS_VLLM_026`):
`_send_unpacked`, `packed` placement, `npu_generate_uuid` arity, and
`start/finish_weight_update` no-op behavior asserted per version |
| `vllm_ascend/worker/v2/spec_decode/dflash/speculator.py` |
[vllm#50000](vllm-project/vllm#50000) —
`_prepare_dflash_inputs_kernel` gained `temperature`/`seeds` sampling
params for probabilistic draft sampling | Version-gated
`_prepare_dflash_inputs_kernel_ascend`: 0.26.0 branch keeps the old
signature; main branch adds the four `temperature`/`seeds` pointers and
the corresponding stores to stay aligned with upstream while keeping
Ascend's own kernel |
| `.github/vllm-main-verified.commit` | — | Updated verified main commit
hash to `0351e9aa1fdf1a51329d1906881528dfe61fc88e` |

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?


- vLLM version: v0.26.0
- vLLM main:
vllm-project/vllm@d02df74

---------

Signed-off-by: hfadzxy <starmoon_zhang@163.com>
MmMmaru pushed a commit to jiaqi-lee/vllm-ascend that referenced this pull request Aug 19, 2026
### What this PR does / why we need it?

| Files | Upstream vLLM change | vllm-ascend adaptation |
|-------|---------------------|------------------------|
| `vllm_ascend/__init__.py` |
[vllm#48841](vllm-project/vllm#48841) — added
`from triton.experimental import gluon`, `from triton.experimental.gluon
import language as gl` and `from triton.language.core import _aggregate`
to `triton_utils/__init__.py`, requiring Triton 3.6+ API absent from
`triton-ascend 3.2.1` | Pre-register `triton.experimental.gluon` /
`.language` as `sys.modules` stubs; stub
`triton.language.core._aggregate`. Gated on `os.getenv("VLLM_VERSION")
!= "0.26.0"`, mirroring `vllm_version_is` env var path. |
| `tests/ut/patch/platform/test_patch_structured_output.py` |
[vllm#49665](vllm-project/vllm#49665) — changed
`VLLMValidationError` base class from `ValueError` to `VLLMClientError`,
breaking `pytest.raises(ValueError)` assertions | Updated 3 assertions
from `pytest.raises(ValueError, ...)` to
`pytest.raises(VLLMValidationError, ...)` |
| `tests/e2e/pull_request/one_card/test_guided_decoding.py` |
[vllm#49665](vllm-project/vllm#49665) — same
`VLLMValidationError` base-class change; on 0.26.0 the upstream
validation may still raise `ValueError` | Version-gated assertion:
`pytest.raises(ValueError, ...)` on 0.26.0 vs
`pytest.raises(VLLMValidationError, ...)` on main |
|
`vllm_ascend/ops/vocab_parallel_embedding.py`<br>`vllm_ascend/_310p/ops/vocab_parallel_embedding.py`
| [vllm#49731](vllm-project/vllm#49731) — added
`*, disable_tp: bool = False` keyword arg to `ParallelLMHead.__init__()`
and `VocabParallelEmbedding.__init__()` | Version-gated `__init__` with
`vllm_version_is("0.26.0")`: else branch accepts `disable_tp` kwarg and
forwards it (Ascend manages TP via `lmhead_tp_enable()`) |
|
`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py`<br>`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_hybrid_connector.py`
| [vllm#26593](vllm-project/vllm#26593) —
`KVConnectorBase_V1.__init__()` sets `self._kv_transfer_config`,
required by `requires_kv_delivery` property | Version-gated `__init__`:
else branch sets `self._kv_transfer_config` on vllm main |
| `vllm_ascend/ops/fused_moe/fused_moe.py` |
[vllm#50089](vllm-project/vllm#50089) — added
`fused_output_is_reduced` kwarg to
`_maybe_reduce_shared_expert_output()` and `output_is_reduced` kwarg to
`_maybe_reduce_final_output()` | Version-gated both methods: else branch
accepts new kwargs but ignores them (Ascend handles reduction
independently); `trunc_size` may now be `None` |
| `vllm_ascend/patch/worker/patch_distributed.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `GroupCoordinator.__init__()` |
Version-gated via `_IS_VLLM_026` env var check (avoids vllm-ascend
import cycle). Else branch stores `self.use_all2all` (unused, no all2all
on Ascend) |
| `vllm_ascend/distributed/device_communicators/npu_communicator.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `DeviceCommunicatorBase.__init__()`
| Version-gated `NPUCommunicator.__init__` with
`vllm_version_is("0.26.0")`: else branch accepts and forwards
`use_all2all` (NPU keeps the no-op `_NpuAll2AllManager`) |
| `vllm_ascend/worker/npu_input_batch.py` |
[vllm#48018](vllm-project/vllm#48018) — added
`use_replayssm: bool` kwarg (Mamba replay-SSM);
[vllm#40996](vllm-project/vllm#40996) — added
`slot_mapping_modes` kwarg (DCP hybrid attention) | Accepts both kwargs
unconditionally; stores `self.use_replayssm` and
`self.slot_mapping_modes` only on main (`not vllm_version_is("0.26.0")`)
for interface alignment |
| `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` |
[vllm#49364](vllm-project/vllm#49364) — renamed
`skip_attn` parameter to `full_cudagraph` in cudagraph mode logic |
Version-gated call: 0.26.0 keeps `skip_attn=(cg_mode != PIECEWISE)`;
main passes `full_cudagraph=(cg_mode != PIECEWISE)` |
|
`vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py`<br>`vllm_ascend/distributed/weight_transfer/__init__.py`
| [vllm#48042](vllm-project/vllm#48042) +
[vllm#48981](vllm-project/vllm#48981) — replaced
static `IPCTrainerSendWeightsArgs`/`trainer_send_weights` with a
stateful `IPCTrainerWeightTransferEngine` driven by
`WeightTransferTrainerFactory.trainer_init(...).send_weights()`;
`packed` moved onto trainer init-info; per-round `update_info` slimmed |
Whole module version-gated: 0.26.0 keeps the static
`NPUIPCWeightTransferEngine`; main defines stateful
`NPUIPCTrainerWeightTransferEngine` (`_send_unpacked` instance method,
`packed` on `NPUIPCWeightTransferInitInfo`, no-arg
`npu_generate_uuid()`, `is_sender`/`_all_gather_and_merge_handles`).
`register_engine()` also registers the trainer engine in
`WeightTransferTrainerFactory` on main only |
| `examples/rl/rlhf_http_npu_ipc.py` |
[vllm#48981](vllm-project/vllm#48981) — ported
RL examples to `WeightTransferTrainerFactory.trainer_init(...)` +
`engine.send_weights()` with `HTTPVLLMWeightSyncClient`/`ModuleSource` |
Version-gated example: 0.26.0 uses static
`NPUIPCWeightTransferEngine.trainer_send_weights`; main uses the
stateful trainer engine path |
| `tests/e2e/pull_request/one_card/test_npu_ipc_weight_transfer.py` |
[vllm#48981](vllm-project/vllm#48981) — IPC
weight-transfer E2E now exercised via the stateful trainer engine |
Version-gated test: 0.26.0 keeps the manual `_post(start/finish)` +
static `trainer_send_weights`; main drives
`WeightTransferTrainerFactory.trainer_init(...).send_weights()` (engine
owns the lifecycle) |
| `tests/ut/distributed/weight_transfer/test_npu_ipc_engine.py` |
[vllm#48981](vllm-project/vllm#48981) —
trainer-side IPC APIs changed from static methods to instance methods on
the stateful engine | Version-gated tests (`IS_VLLM_026`):
`_send_unpacked`, `packed` placement, `npu_generate_uuid` arity, and
`start/finish_weight_update` no-op behavior asserted per version |
| `vllm_ascend/worker/v2/spec_decode/dflash/speculator.py` |
[vllm#50000](vllm-project/vllm#50000) —
`_prepare_dflash_inputs_kernel` gained `temperature`/`seeds` sampling
params for probabilistic draft sampling | Version-gated
`_prepare_dflash_inputs_kernel_ascend`: 0.26.0 branch keeps the old
signature; main branch adds the four `temperature`/`seeds` pointers and
the corresponding stores to stay aligned with upstream while keeping
Ascend's own kernel |
| `.github/vllm-main-verified.commit` | — | Updated verified main commit
hash to `0351e9aa1fdf1a51329d1906881528dfe61fc88e` |

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?


- vLLM version: v0.26.0
- vLLM main:
vllm-project/vllm@d02df74

---------

Signed-off-by: hfadzxy <starmoon_zhang@163.com>
shiqiangA pushed a commit to shiqiangA/vllm-ascend that referenced this pull request Aug 20, 2026
### What this PR does / why we need it?

| Files | Upstream vLLM change | vllm-ascend adaptation |
|-------|---------------------|------------------------|
| `vllm_ascend/__init__.py` |
[vllm#48841](vllm-project/vllm#48841) — added
`from triton.experimental import gluon`, `from triton.experimental.gluon
import language as gl` and `from triton.language.core import _aggregate`
to `triton_utils/__init__.py`, requiring Triton 3.6+ API absent from
`triton-ascend 3.2.1` | Pre-register `triton.experimental.gluon` /
`.language` as `sys.modules` stubs; stub
`triton.language.core._aggregate`. Gated on `os.getenv("VLLM_VERSION")
!= "0.26.0"`, mirroring `vllm_version_is` env var path. |
| `tests/ut/patch/platform/test_patch_structured_output.py` |
[vllm#49665](vllm-project/vllm#49665) — changed
`VLLMValidationError` base class from `ValueError` to `VLLMClientError`,
breaking `pytest.raises(ValueError)` assertions | Updated 3 assertions
from `pytest.raises(ValueError, ...)` to
`pytest.raises(VLLMValidationError, ...)` |
| `tests/e2e/pull_request/one_card/test_guided_decoding.py` |
[vllm#49665](vllm-project/vllm#49665) — same
`VLLMValidationError` base-class change; on 0.26.0 the upstream
validation may still raise `ValueError` | Version-gated assertion:
`pytest.raises(ValueError, ...)` on 0.26.0 vs
`pytest.raises(VLLMValidationError, ...)` on main |
|
`vllm_ascend/ops/vocab_parallel_embedding.py`<br>`vllm_ascend/_310p/ops/vocab_parallel_embedding.py`
| [vllm#49731](vllm-project/vllm#49731) — added
`*, disable_tp: bool = False` keyword arg to `ParallelLMHead.__init__()`
and `VocabParallelEmbedding.__init__()` | Version-gated `__init__` with
`vllm_version_is("0.26.0")`: else branch accepts `disable_tp` kwarg and
forwards it (Ascend manages TP via `lmhead_tp_enable()`) |
|
`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py`<br>`vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_hybrid_connector.py`
| [vllm#26593](vllm-project/vllm#26593) —
`KVConnectorBase_V1.__init__()` sets `self._kv_transfer_config`,
required by `requires_kv_delivery` property | Version-gated `__init__`:
else branch sets `self._kv_transfer_config` on vllm main |
| `vllm_ascend/ops/fused_moe/fused_moe.py` |
[vllm#50089](vllm-project/vllm#50089) — added
`fused_output_is_reduced` kwarg to
`_maybe_reduce_shared_expert_output()` and `output_is_reduced` kwarg to
`_maybe_reduce_final_output()` | Version-gated both methods: else branch
accepts new kwargs but ignores them (Ascend handles reduction
independently); `trunc_size` may now be `None` |
| `vllm_ascend/patch/worker/patch_distributed.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `GroupCoordinator.__init__()` |
Version-gated via `_IS_VLLM_026` env var check (avoids vllm-ascend
import cycle). Else branch stores `self.use_all2all` (unused, no all2all
on Ascend) |
| `vllm_ascend/distributed/device_communicators/npu_communicator.py` |
[vllm#47288](vllm-project/vllm#47288) — added
`use_all2all: bool = False` param to `DeviceCommunicatorBase.__init__()`
| Version-gated `NPUCommunicator.__init__` with
`vllm_version_is("0.26.0")`: else branch accepts and forwards
`use_all2all` (NPU keeps the no-op `_NpuAll2AllManager`) |
| `vllm_ascend/worker/npu_input_batch.py` |
[vllm#48018](vllm-project/vllm#48018) — added
`use_replayssm: bool` kwarg (Mamba replay-SSM);
[vllm#40996](vllm-project/vllm#40996) — added
`slot_mapping_modes` kwarg (DCP hybrid attention) | Accepts both kwargs
unconditionally; stores `self.use_replayssm` and
`self.slot_mapping_modes` only on main (`not vllm_version_is("0.26.0")`)
for interface alignment |
| `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` |
[vllm#49364](vllm-project/vllm#49364) — renamed
`skip_attn` parameter to `full_cudagraph` in cudagraph mode logic |
Version-gated call: 0.26.0 keeps `skip_attn=(cg_mode != PIECEWISE)`;
main passes `full_cudagraph=(cg_mode != PIECEWISE)` |
|
`vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py`<br>`vllm_ascend/distributed/weight_transfer/__init__.py`
| [vllm#48042](vllm-project/vllm#48042) +
[vllm#48981](vllm-project/vllm#48981) — replaced
static `IPCTrainerSendWeightsArgs`/`trainer_send_weights` with a
stateful `IPCTrainerWeightTransferEngine` driven by
`WeightTransferTrainerFactory.trainer_init(...).send_weights()`;
`packed` moved onto trainer init-info; per-round `update_info` slimmed |
Whole module version-gated: 0.26.0 keeps the static
`NPUIPCWeightTransferEngine`; main defines stateful
`NPUIPCTrainerWeightTransferEngine` (`_send_unpacked` instance method,
`packed` on `NPUIPCWeightTransferInitInfo`, no-arg
`npu_generate_uuid()`, `is_sender`/`_all_gather_and_merge_handles`).
`register_engine()` also registers the trainer engine in
`WeightTransferTrainerFactory` on main only |
| `examples/rl/rlhf_http_npu_ipc.py` |
[vllm#48981](vllm-project/vllm#48981) — ported
RL examples to `WeightTransferTrainerFactory.trainer_init(...)` +
`engine.send_weights()` with `HTTPVLLMWeightSyncClient`/`ModuleSource` |
Version-gated example: 0.26.0 uses static
`NPUIPCWeightTransferEngine.trainer_send_weights`; main uses the
stateful trainer engine path |
| `tests/e2e/pull_request/one_card/test_npu_ipc_weight_transfer.py` |
[vllm#48981](vllm-project/vllm#48981) — IPC
weight-transfer E2E now exercised via the stateful trainer engine |
Version-gated test: 0.26.0 keeps the manual `_post(start/finish)` +
static `trainer_send_weights`; main drives
`WeightTransferTrainerFactory.trainer_init(...).send_weights()` (engine
owns the lifecycle) |
| `tests/ut/distributed/weight_transfer/test_npu_ipc_engine.py` |
[vllm#48981](vllm-project/vllm#48981) —
trainer-side IPC APIs changed from static methods to instance methods on
the stateful engine | Version-gated tests (`IS_VLLM_026`):
`_send_unpacked`, `packed` placement, `npu_generate_uuid` arity, and
`start/finish_weight_update` no-op behavior asserted per version |
| `vllm_ascend/worker/v2/spec_decode/dflash/speculator.py` |
[vllm#50000](vllm-project/vllm#50000) —
`_prepare_dflash_inputs_kernel` gained `temperature`/`seeds` sampling
params for probabilistic draft sampling | Version-gated
`_prepare_dflash_inputs_kernel_ascend`: 0.26.0 branch keeps the old
signature; main branch adds the four `temperature`/`seeds` pointers and
the corresponding stores to stay aligned with upstream while keeping
Ascend's own kernel |
| `.github/vllm-main-verified.commit` | — | Updated verified main commit
hash to `0351e9aa1fdf1a51329d1906881528dfe61fc88e` |

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?


- vLLM version: v0.26.0
- vLLM main:
vllm-project/vllm@d02df74

---------

Signed-off-by: hfadzxy <starmoon_zhang@163.com>
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends frontend intel-gpu Related to Intel GPU nvidia ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

7 participants