Skip to content

[AMD][Bugfix][EPLB] Fix elastic EP scaling accuracy on ROCm - #47206

Merged
vllm-bot merged 12 commits into
vllm-project:mainfrom
okorzh-amd:fix/elastic-ep-scaling-rocm
Jul 24, 2026
Merged

vllm-bot merged 12 commits into
vllm-project:mainfrom
okorzh-amd:fix/elastic-ep-scaling-rocm

Conversation

@okorzh-amd

@okorzh-amd okorzh-amd commented Jun 30, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Two fixes for distributed/test_elastic_ep.py, which collapsed GSM8K accuracy at the initial and scaled checkpoints on ROCm:

Bug 1 Scale-up produces garbage output on ROCm (accuracy=0)

Root cause. New elastic-EP ranks are created with load_dummy_weights=True. On ROCm, initialize_single_dummy_weight() .zero_()s every non-floating-point tensor in model.state_dict(). The FusedMoE expert-topology buffers expert_map, expert_mask, and expert{global_to_physical,physical_to_global,local_to_global} were registered as persistent integer buffers, so they were zeroed right after being built correctly for the new EP topology. That corrupted all2all dispatch garbage output. CUDA never zeroes integer tensors, so the bug was ROCm-only.

This corrects an earlier hypothesis: the map is not built unsharded at ep_size==1; it is built correctly at the target ep_size and then corrupted. (Thanks @itayalroy for identifying the dummy-init zeroing path.)

Fix. Register those five buffers with persistent=False. They are per-rank, deterministically-derived EP topology metadata not checkpoint state so they should never be in state_dict() (matches moondream3.py's _expert_map registration). This keeps them out of state_dict(), so:

  • ROCm dummy-weight init can no longer zero them, and
  • batch_transfer_weights() (which iterates state_dict()) can no longer clobber them from a sender rank.

As a result, no post-reshuffle metadata rebuild is needed and no name-based transfer filter for these tensors is needed.

Bug 2 Sync-EPLB accuracy loss from redundant rearranges

rearrange_expert_weights_inplace() was firing every step_interval even when the proposed map barely improved balance (imbalance 1.0), causing ~16s lateral all-to-all churn that starved the engine and dropped accuracy. We now compute per-rank load imbalance for the current vs proposed map and skip the rearrange when improvement is <5%, decided collectively via all-reduce and guarded to non-scale-up rearranges.

Test Plan

run failed CI tests

Test Result

distributed/test_elastic_ep.py::test_elastic_ep_scaling[sync_eplb]          PASSED
distributed/test_elastic_ep.py::test_elastic_ep_scaling[async_eplb]         PASSED
distributed/test_elastic_ep.py::test_elastic_ep_scaling_uneven[sync_eplb]   PASSED
distributed/test_elastic_ep.py::test_elastic_ep_scaling_uneven[async_eplb]  PASSED
================= 4 passed, 18 warnings in 1194.63s (0:19:54) ==================

Per-checkpoint accuracies (all >= 0.58 threshold; previously 0.19 / 0.016):

image
Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added rocm Related to AMD ROCm bug Something isn't working labels Jun 30, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Jun 30, 2026
@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from 68dacf1 to e36e155 Compare June 30, 2026 22:06
@AndreasKaratzas AndreasKaratzas added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 30, 2026
@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from e36e155 to d2a84dd Compare June 30, 2026 22:57
Comment thread vllm/distributed/eplb/eplb_state.py Outdated
Comment on lines +855 to +880
_old_imb = _rank_imbalance(
eplb_model_state.physical_to_logical_map, _gl, _ep
)
_new_imb = _rank_imbalance(new_physical_to_logical_map, _gl, _ep)
if _old_imb <= 1e-6 or (_old_imb - _new_imb) / _old_imb < 0.05:
skip_rearrange = True
if is_main_rank:
logger.info(
"[EPLB] Skip rearrange: imbalance %.4f -> "
"%.4f (no material gain)",
_old_imb,
_new_imb,
)

if not skip_rearrange:
# Update expert weights
rearrange_expert_weights_inplace(
eplb_model_state.physical_to_logical_map,
new_physical_to_logical_map,
eplb_model_state.model.expert_weights,
eplb_model_state.expert_buffer,
ep_group,
eplb_model_state.communicator,
is_profile,
rank_mapping,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this cause deadlocks if one rank decides to rearrange and another decides not to, due to slight numerical differences when calculating the imbalance?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current design the decision cannot actually diverge, because every input to the skip test is rank-identical:

  • global_expert_load_window (_gl) is the all-reduced load, so it's the same tensor on every rank.
  • physical_to_logical_map is a replicated global invariant, and new_physical_to_logical_map comes from the deterministic policy.rebalance_experts(...) fed that same all-reduced load so it's identical on every rank too.
  • _rank_imbalance is a pure function of those integer maps + scalars. Same inputs + same op sequence on the same device type leads to bit-identical float result. FP "slight numerical differences" only appear when the inputs differ across ranks, which they don't here.

So on a homogeneous cluster there's no split-brain. But that safety is an implicit invariant, and it would silently break on a heterogeneous cluster (mixed GPU archs) or if someone later feeds a non-all-reduced load. Since rearrange_expert_weights_inplace is a collective, a divergent decision would deadlock. Let me try a solution with reduce to see how much perf it is going to cost.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perf looks good. I've update the code to have all_reduce for the decision

@tlrmchlsmth

Copy link
Copy Markdown
Member

~16-21s of tiny per-expert P2P

Wow that is very significant! What library are you using for EPLB transfers?

Comment on lines +554 to +562
if rank_mapping is None: # scale-up
model = self.worker.model_runner.get_model()
if hasattr(model, "update_physical_experts_metadata"):
ep_size = get_ep_group().world_size
num_physical_experts = eplb_model_state.physical_to_logical_map.shape[1]
model.update_physical_experts_metadata(
num_physical_experts=num_physical_experts,
num_local_physical_experts=num_physical_experts // ep_size,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed on on RoCM? Or is it a general bug?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it is shared path.
@pavanimajety who would be a good PIC from Nvidia to review it?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itayalroy would be the right one

@itayalroy itayalroy Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this metadata update needed? both EEP reconfiguration and EPLB reshuffle never change which physical slots each rank owns, new ranks should already have the correct metadata.

I think the issue actually comes from batch_transfer_weights(), it overwrites the new rank’s AITER expert_mask with expert_mask from an existing rank, because it excludes expert_map by name but it does not exclude expert_mask (and this is why we don't see an issue with CUDA btw). Can we try to exclude the ROCM expert metadata there as well?

We should really avoid deciding what to transfer based on tensor names and explicitly define which model state should be copied to the new workers, this batch_transfer_weights filter is very hacky, but since this is existing behavior we can do it in a separate PR (unless you are up to fixing it)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the expert_mask filter gap. I've added it to the exclusion (it's genuinely per-rank metadata that shouldn't be transferred). But I tested dropping the update_physical_experts_metadata rebuild and relying only on the exclusion, and scale-up still produces garbage (GSM8K 0.008). The root cause isn't the clobber: new scale-up ranks build their FusedMoE before joining the new EP group, so determine_expert_map() runs at ep_size==1 and returns an unsharded map (every global expert local) which matches the "all-local" symptom, not a sender's shard. So the rebuild is doing the real work of resharding for the new topology; the expert_mask exclusion is a correctness/hygiene fix on top. All 4 test_elastic_ep.py params pass on MI355X.

@itayalroy itayalroy Jul 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New ranks start with the target DP size (code). FusedMoE reads that size from the DP group (code), calculates flattened TP size as DP × PCP × TP (code), and uses that value as ep_size (code). This ep_size is passed to determine_expert_map() (code), so this value is never 1 for new ranks (unless ofc target ep size is actually 1). Moreover, if new ranks were initialized with ep_size == 1, we would also see the problem on CUDA because this path is shared. Finally I also printed ep_size from determine_expert_map() on new ranks and it equals the target ep size, so new ranks initializing FusedMoE with ep_size == 1 is definitely not the RC here.

I think that with ROCm, the expert metadata is getting corrupted somewhere along the way. We already found one issue in batch weight transfer. Another likely issue is during model loading: new ranks load model with dummy weights (code), and specifically on ROCm this seems to zero every non-floating-point tensor in state_dict() (code). Since _expert_map and expert_mask are registered integer buffers, they are probably being zeroed there even though they shouldn't be (they are metadata, not weights). The CUDA path does not zero integer tensors, which would explain why we do not see the same issue there.

There may be other issues as well, but I think we should identify the actual RC rather than rely on this update expert metadata call without fully understanding what went wrong. Otherwise, now or in the future, other tensors (unrelated to expert metadata) may get corrupted by the same path and not repaired.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itayalroy thanks for your comments. I've updated the code and PR description. Please review when you have time.

@okorzh-amd

Copy link
Copy Markdown
Contributor Author

Wow that is very significant! What library are you using for EPLB transfers?

@tlrmchlsmth torch_nccl is silently promoted to pynccl (see eplb_communicator.py ~L739-744) which actually is RCCL ncclSend/ncclRecv for the test.

@tlrmchlsmth

Copy link
Copy Markdown
Member

Wow that is very significant! What library are you using for EPLB transfers?

@tlrmchlsmth torch_nccl is silently promoted to pynccl (see eplb_communicator.py ~L739-744) which actually is RCCL ncclSend/ncclRecv for the test.

I see - I asked because we switched the default EPLB communication library to be NIXL with a fallback to gloo. We found that NCCL can sometimes hang (due to a NCCL bug that was subsequently fixed), and NCCL is not compatible with the async case due to a lack of forward progress guarantees in CUDA. Is this the case in RCCL/ROCm as well?

@okorzh-amd

Copy link
Copy Markdown
Contributor Author

Wow that is very significant! What library are you using for EPLB transfers?

@tlrmchlsmth torch_nccl is silently promoted to pynccl (see eplb_communicator.py ~L739-744) which actually is RCCL ncclSend/ncclRecv for the test.

I see - I asked because we switched the default EPLB communication library to be NIXL with a fallback to gloo. We found that NCCL can sometimes hang (due to a NCCL bug that was subsequently fixed), and NCCL is not compatible with the async case due to a lack of forward progress guarantees in CUDA. Is this the case in RCCL/ROCm as well?

it's a small-message P2P performance problem, not the NCCL hang bug. So the answer to your question is yes, but this particular code is platform-agnostic.

@SageMoore

Copy link
Copy Markdown
Contributor

each sync rearrange is ~16-21s

@okorzh-amd this is in incredible amount of transfer time. Can you elaborate on exactly what workload gives you those times? Are they exclusive to multi-node or are you seeing those transfer times on single node as well? I'd be surprised if EPLB gives any performance boost with transfer times like that.

@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from f15529f to 1ae234a Compare July 13, 2026 14:10
okorzh-amd added a commit to okorzh-amd/vllm that referenced this pull request Jul 13, 2026
All-reduce the per-rank skip vote so the decision to skip a no-benefit
rearrange is provably unanimous across the EP group. Each rank votes
locally on the imbalance threshold; skip only when every rank agrees,
otherwise fall back to the always-correct rearrange. This removes any
risk of a split-brain (some ranks entering the collective
rearrange_expert_weights_inplace while others skip) deadlocking, even
under numerical drift on heterogeneous hardware.

Addresses review feedback on vllm-project#47206.

Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
okorzh-amd added a commit to okorzh-amd/vllm that referenced this pull request Jul 13, 2026
batch_transfer_weights() filters transferred tensors by name and only
skipped `expert_map`. On ROCm/AITER the per-rank routing metadata also lives
in the separate `expert_mask` buffer, which was therefore received from the
source rank; excluding it keeps each worker's own metadata intact (CUDA was
unaffected since it uses `_expert_map` and `expert_mask` is None there).

Also document why the update_physical_experts_metadata rebuild is still
required: new scale-up ranks build their FusedMoE before joining the new EP
group, so their expert_map/expert_mask are unsharded (ep_size==1, every
expert marked local) until rebuilt for the new topology. Validated on MI355X:
all 4 test_elastic_ep.py params pass, GSM8K ~0.64-0.68 at every checkpoint.

Addresses review feedback on vllm-project#47206.

Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from d82c7ee to e8849fd Compare July 22, 2026 02:19
@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from e8849fd to 81b090e Compare July 22, 2026 03:08
@okorzh-amd

Copy link
Copy Markdown
Contributor Author

@okorzh-amd this is in incredible amount of transfer time. Can you elaborate on exactly what workload gives you those times? Are they exclusive to multi-node or are you seeing those transfer times on single node as well? I'd be surprised if EPLB gives any performance boost with transfer times like that.

@SageMoore good question. These are single-node numbers, so it's not a multi-node fabric issue. Exact workload is the CI test itself (tests/distributed/test_elastic_ep.py::test_elastic_ep_scaling[sync_eplb]): DeepSeek-V2-Lite-Chat on one MI355X node, 4 GPUs, TP=1, DP=2, --enable-eplb sync mode, step_interval=10, window_size=5, num_redundant_experts=0. 26 MoE layers * 64 experts = 1664 physical expert slots.

@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from 81b090e to a93ea35 Compare July 22, 2026 03:39
@okorzh-amd
okorzh-amd requested a review from itayalroy July 22, 2026 03:44

@AndreasKaratzas AndreasKaratzas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tlrmchlsmth PTAL

Comment thread vllm/distributed/eplb/eplb_state.py Outdated
Comment on lines +835 to +868
_gl = global_expert_load_window
_ep = ep_group.size()

def _rank_imbalance(
_p2l: torch.Tensor, _gl: torch.Tensor, _ep: int
) -> float:
_p2l = _p2l.to(_gl.device)
_pl = torch.gather(_gl, 1, _p2l.clamp(min=0).long())
_pl = _pl * (_p2l >= 0)
_P = _p2l.shape[1]
_per_rank = (
_pl.reshape(_pl.shape[0], _ep, _P // _ep)
.sum(dim=(0, 2))
.float()
)
_mean = _per_rank.mean()
if _mean <= 0:
return 1.0
return (_per_rank.max() / _mean).item()

_old_imb = _rank_imbalance(
eplb_model_state.physical_to_logical_map, _gl, _ep
)
_new_imb = _rank_imbalance(new_physical_to_logical_map, _gl, _ep)
local_skip = (
_old_imb <= 1e-6 or (_old_imb - _new_imb) / _old_imb < 0.05
)
_vote = torch.tensor(
[1 if local_skip else 0],
device=_gl.device,
dtype=torch.int32,
)
all_reduce(_vote, group=ep_group)
skip_rearrange = int(_vote.item()) == _ep

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_gl = global_expert_load_window
_ep = ep_group.size()
def _rank_imbalance(
_p2l: torch.Tensor, _gl: torch.Tensor, _ep: int
) -> float:
_p2l = _p2l.to(_gl.device)
_pl = torch.gather(_gl, 1, _p2l.clamp(min=0).long())
_pl = _pl * (_p2l >= 0)
_P = _p2l.shape[1]
_per_rank = (
_pl.reshape(_pl.shape[0], _ep, _P // _ep)
.sum(dim=(0, 2))
.float()
)
_mean = _per_rank.mean()
if _mean <= 0:
return 1.0
return (_per_rank.max() / _mean).item()
_old_imb = _rank_imbalance(
eplb_model_state.physical_to_logical_map, _gl, _ep
)
_new_imb = _rank_imbalance(new_physical_to_logical_map, _gl, _ep)
local_skip = (
_old_imb <= 1e-6 or (_old_imb - _new_imb) / _old_imb < 0.05
)
_vote = torch.tensor(
[1 if local_skip else 0],
device=_gl.device,
dtype=torch.int32,
)
all_reduce(_vote, group=ep_group)
skip_rearrange = int(_vote.item()) == _ep
logical_loads = global_expert_load_window.float()
ep_size = ep_group.size()
def rank_load_imbalance(mapping: torch.Tensor) -> float:
mapping = mapping.to(
device=logical_loads.device,
dtype=torch.long,
)
replica_counts = torch.zeros_like(logical_loads)
replica_counts.scatter_add_(
dim=1,
index=mapping,
src=torch.ones_like(mapping, dtype=logical_loads.dtype),
)
loads_per_replica = torch.gather(
logical_loads / replica_counts.clamp_min(1),
dim=1,
index=mapping,
)
loads_per_rank = loads_per_replica.reshape(
logical_loads.shape[0], ep_size, -1
).sum(dim=(0, 2))
mean_load = loads_per_rank.mean()
if mean_load == 0:
return 1.0
return (loads_per_rank.max() / mean_load).item()
current_imbalance = rank_load_imbalance(
eplb_model_state.physical_to_logical_map
)
proposed_imbalance = rank_load_imbalance(
new_physical_to_logical_map
)
relative_improvement = (
current_imbalance - proposed_imbalance
) / current_imbalance
skip_rearrange = relative_improvement < 0.05

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to simplify + clean up this block of code before landing - @okorzh-amd and @AndreasKaratzas could you take a look?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should conservatively skip rearranging on non-AMD hardware as well IMO, since it adds an allreduce.

okorzh-amd and others added 3 commits July 22, 2026 20:48
Adopt @tlrmchlsmth's suggestion for the low-gain rearrange skip check
(PR vllm-project#47206, r3634076167):

- rank_load_imbalance() now splits each logical expert's load across its
  physical replicas (scatter_add replica_counts, divide load) so the
  imbalance estimate matches how runtime traffic actually distributes.
  Behavior is identical when num_redundant_experts=0 (the test config) and
  strictly more accurate once experts are replicated.
- Drop the per-rank all_reduce consensus vote: global_expert_load_window is
  already all-reduced and policy.rebalance_experts is deterministic, so every
  rank derives the same decision without an extra collective.
- Remove now-dead guards (imbalance = max/mean is always >= 1.0) and the
  clamp/mask (the outer map >= 0 guard covers it).

Gate the whole skip path behind current_platform.is_rocm(): the redundant-
rearrange accuracy collapse this guards against was only observed on ROCm;
other platforms keep the unconditional rearrange behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: okorzh <okorzh@amd.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: okorzh <okorzh@amd.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: okorzh <okorzh@amd.com>
@okorzh-amd
okorzh-amd force-pushed the fix/elastic-ep-scaling-rocm branch from 558e33c to 908e774 Compare July 23, 2026 01:48
@mergify

mergify Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Hi @okorzh-amd, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

okorzh-amd and others added 2 commits July 22, 2026 21:03
rank_load_imbalance() is defined inside the per-model loop and closed over
the loop-scoped logical_loads / ep_size, tripping ruff B023. Bind them as
default arguments (evaluated at def time) so the closure captures the
current iteration's values.

Signed-off-by: okorzh <okorzh@amd.com>
@mergify

mergify Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Hi @okorzh-amd, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@mergify

mergify Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Hi @okorzh-amd, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@vllm-bot
vllm-bot merged commit 2279575 into vllm-project:main Jul 24, 2026
120 of 125 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Jul 24, 2026
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
…ject#47206)

Signed-off-by: okorzh <okorzh@amd.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
deveringham added a commit to deveringham/vllm-ascend that referenced this pull request Oct 20, 2026
Each policy returns (changed, per_layer_priority, new_placement), and
policy_abstract documents the first value as 1 = table_changed, 0 =
not_changed. EplbWorker.do_update drops it:

    _, _, new_placement = self.calculate_rebalance_experts(...)

so every plan is applied, including ones the policy itself judged not worth
applying. In DefaultEplb and SwiftBalanceEplb the flag is 1 only when the plan
cuts the summed per-layer peak load by at least 5%; a plan below that bar
still moves expert weights between NPUs for no gain.

This reads the flag and keeps the current placement when it is 0. That needs
no new control flow: every layer then compares equal in
compose_expert_update_info_greedy, which yields empty send and receive sets,
and the loader completes its cycle without moving anything. That path already
runs today for any layer whose map does not change. Returning early would not
be safe, because forward_before calls pop(0) on whatever the worker queues.

The decision is the same on every rank: load_info is the all-gathered load and
the policy and this check are deterministic, so no rank can diverge.

The same path also replaces the early return when no load is available. That
return queues None, which forward_before stores and the next pop(0) turns into
an AttributeError. Nothing reaches it today because warm_up_eplb always
gathers a load first, but keeping the placement removes the latent crash.

There is precedent upstream: vLLM core skips a rearrangement when the relative
improvement in imbalance is under 5% (vllm/distributed/eplb/eplb_state.py,
added in vllm-project/vllm#47206, currently ROCm only).

per_layer_priority is still discarded. It would matter if the number of layers
moved per cycle were ever capped; this leaves it alone.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Dylan Everingham <dceveringham@gmail.com>
deveringham added a commit to deveringham/vllm-ascend that referenced this pull request Oct 21, 2026
Each policy returns (changed, per_layer_priority, new_placement), and
policy_abstract documents the first value as 1 = table_changed, 0 =
not_changed. EplbWorker.do_update drops it:

    _, _, new_placement = self.calculate_rebalance_experts(...)

so every plan is applied, including ones the policy itself judged not worth
applying. In DefaultEplb and SwiftBalanceEplb the flag is 1 only when the plan
cuts the summed per-layer peak load by at least 5%; a plan below that bar
still moves expert weights between NPUs for no gain.

This reads the flag and keeps the current placement when it is 0. That needs
no new control flow: every layer then compares equal in
compose_expert_update_info_greedy, which yields empty send and receive sets,
and the loader completes its cycle without moving anything. That path already
runs today for any layer whose map does not change. Returning early would not
be safe, because forward_before calls pop(0) on whatever the worker queues.

The decision is the same on every rank: load_info is the all-gathered load and
the policy and this check are deterministic, so no rank can diverge.

The same path also replaces the early return when no load is available. That
return queues None, which forward_before stores and the next pop(0) turns into
an AttributeError. Nothing reaches it today because warm_up_eplb always
gathers a load first, but keeping the placement removes the latent crash.

There is precedent upstream: vLLM core skips a rearrangement when the relative
improvement in imbalance is under 5% (vllm/distributed/eplb/eplb_state.py,
added in vllm-project/vllm#47206, currently ROCm only).

per_layer_priority is still discarded. It would matter if the number of layers
moved per cycle were ever capped; this leaves it alone.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Dylan Everingham <dceveringham@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

7 participants