Repository navigation
[AMD][Bugfix][EPLB] Fix elastic EP scaling accuracy on ROCm - #47206
Conversation
68dacf1 to
e36e155
Compare
e36e155 to
d2a84dd
Compare
| _old_imb = _rank_imbalance( | ||
| eplb_model_state.physical_to_logical_map, _gl, _ep | ||
| ) | ||
| _new_imb = _rank_imbalance(new_physical_to_logical_map, _gl, _ep) | ||
| if _old_imb <= 1e-6 or (_old_imb - _new_imb) / _old_imb < 0.05: | ||
| skip_rearrange = True | ||
| if is_main_rank: | ||
| logger.info( | ||
| "[EPLB] Skip rearrange: imbalance %.4f -> " | ||
| "%.4f (no material gain)", | ||
| _old_imb, | ||
| _new_imb, | ||
| ) | ||
|
|
||
| if not skip_rearrange: | ||
| # Update expert weights | ||
| rearrange_expert_weights_inplace( | ||
| eplb_model_state.physical_to_logical_map, | ||
| new_physical_to_logical_map, | ||
| eplb_model_state.model.expert_weights, | ||
| eplb_model_state.expert_buffer, | ||
| ep_group, | ||
| eplb_model_state.communicator, | ||
| is_profile, | ||
| rank_mapping, | ||
| ) |
There was a problem hiding this comment.
Can this cause deadlocks if one rank decides to rearrange and another decides not to, due to slight numerical differences when calculating the imbalance?
There was a problem hiding this comment.
In the current design the decision cannot actually diverge, because every input to the skip test is rank-identical:
- global_expert_load_window (_gl) is the all-reduced load, so it's the same tensor on every rank.
- physical_to_logical_map is a replicated global invariant, and new_physical_to_logical_map comes from the deterministic policy.rebalance_experts(...) fed that same all-reduced load so it's identical on every rank too.
- _rank_imbalance is a pure function of those integer maps + scalars. Same inputs + same op sequence on the same device type leads to bit-identical float result. FP "slight numerical differences" only appear when the inputs differ across ranks, which they don't here.
So on a homogeneous cluster there's no split-brain. But that safety is an implicit invariant, and it would silently break on a heterogeneous cluster (mixed GPU archs) or if someone later feeds a non-all-reduced load. Since rearrange_expert_weights_inplace is a collective, a divergent decision would deadlock. Let me try a solution with reduce to see how much perf it is going to cost.
There was a problem hiding this comment.
perf looks good. I've update the code to have all_reduce for the decision
Wow that is very significant! What library are you using for EPLB transfers? |
| if rank_mapping is None: # scale-up | ||
| model = self.worker.model_runner.get_model() | ||
| if hasattr(model, "update_physical_experts_metadata"): | ||
| ep_size = get_ep_group().world_size | ||
| num_physical_experts = eplb_model_state.physical_to_logical_map.shape[1] | ||
| model.update_physical_experts_metadata( | ||
| num_physical_experts=num_physical_experts, | ||
| num_local_physical_experts=num_physical_experts // ep_size, | ||
| ) |
There was a problem hiding this comment.
Why is this needed on on RoCM? Or is it a general bug?
There was a problem hiding this comment.
yes, it is shared path.
@pavanimajety who would be a good PIC from Nvidia to review it?
There was a problem hiding this comment.
Why is this metadata update needed? both EEP reconfiguration and EPLB reshuffle never change which physical slots each rank owns, new ranks should already have the correct metadata.
I think the issue actually comes from batch_transfer_weights(), it overwrites the new rank’s AITER expert_mask with expert_mask from an existing rank, because it excludes expert_map by name but it does not exclude expert_mask (and this is why we don't see an issue with CUDA btw). Can we try to exclude the ROCM expert metadata there as well?
We should really avoid deciding what to transfer based on tensor names and explicitly define which model state should be copied to the new workers, this batch_transfer_weights filter is very hacky, but since this is existing behavior we can do it in a separate PR (unless you are up to fixing it)
There was a problem hiding this comment.
Good catch on the expert_mask filter gap. I've added it to the exclusion (it's genuinely per-rank metadata that shouldn't be transferred). But I tested dropping the update_physical_experts_metadata rebuild and relying only on the exclusion, and scale-up still produces garbage (GSM8K 0.008). The root cause isn't the clobber: new scale-up ranks build their FusedMoE before joining the new EP group, so determine_expert_map() runs at ep_size==1 and returns an unsharded map (every global expert local) which matches the "all-local" symptom, not a sender's shard. So the rebuild is doing the real work of resharding for the new topology; the expert_mask exclusion is a correctness/hygiene fix on top. All 4 test_elastic_ep.py params pass on MI355X.
There was a problem hiding this comment.
New ranks start with the target DP size (code). FusedMoE reads that size from the DP group (code), calculates flattened TP size as DP × PCP × TP (code), and uses that value as ep_size (code). This ep_size is passed to determine_expert_map() (code), so this value is never 1 for new ranks (unless ofc target ep size is actually 1). Moreover, if new ranks were initialized with ep_size == 1, we would also see the problem on CUDA because this path is shared. Finally I also printed ep_size from determine_expert_map() on new ranks and it equals the target ep size, so new ranks initializing FusedMoE with ep_size == 1 is definitely not the RC here.
I think that with ROCm, the expert metadata is getting corrupted somewhere along the way. We already found one issue in batch weight transfer. Another likely issue is during model loading: new ranks load model with dummy weights (code), and specifically on ROCm this seems to zero every non-floating-point tensor in state_dict() (code). Since _expert_map and expert_mask are registered integer buffers, they are probably being zeroed there even though they shouldn't be (they are metadata, not weights). The CUDA path does not zero integer tensors, which would explain why we do not see the same issue there.
There may be other issues as well, but I think we should identify the actual RC rather than rely on this update expert metadata call without fully understanding what went wrong. Otherwise, now or in the future, other tensors (unrelated to expert metadata) may get corrupted by the same path and not repaired.
There was a problem hiding this comment.
@itayalroy thanks for your comments. I've updated the code and PR description. Please review when you have time.
@tlrmchlsmth torch_nccl is silently promoted to pynccl (see eplb_communicator.py ~L739-744) which actually is RCCL ncclSend/ncclRecv for the test. |
I see - I asked because we switched the default EPLB communication library to be NIXL with a fallback to gloo. We found that NCCL can sometimes hang (due to a NCCL bug that was subsequently fixed), and NCCL is not compatible with the async case due to a lack of forward progress guarantees in CUDA. Is this the case in RCCL/ROCm as well? |
it's a small-message P2P performance problem, not the NCCL hang bug. So the answer to your question is yes, but this particular code is platform-agnostic. |
@okorzh-amd this is in incredible amount of transfer time. Can you elaborate on exactly what workload gives you those times? Are they exclusive to multi-node or are you seeing those transfer times on single node as well? I'd be surprised if EPLB gives any performance boost with transfer times like that. |
f15529f to
1ae234a
Compare
All-reduce the per-rank skip vote so the decision to skip a no-benefit rearrange is provably unanimous across the EP group. Each rank votes locally on the imbalance threshold; skip only when every rank agrees, otherwise fall back to the always-correct rearrange. This removes any risk of a split-brain (some ranks entering the collective rearrange_expert_weights_inplace while others skip) deadlocking, even under numerical drift on heterogeneous hardware. Addresses review feedback on vllm-project#47206. Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
batch_transfer_weights() filters transferred tensors by name and only skipped `expert_map`. On ROCm/AITER the per-rank routing metadata also lives in the separate `expert_mask` buffer, which was therefore received from the source rank; excluding it keeps each worker's own metadata intact (CUDA was unaffected since it uses `_expert_map` and `expert_mask` is None there). Also document why the update_physical_experts_metadata rebuild is still required: new scale-up ranks build their FusedMoE before joining the new EP group, so their expert_map/expert_mask are unsharded (ep_size==1, every expert marked local) until rebuilt for the new topology. Validated on MI355X: all 4 test_elastic_ep.py params pass, GSM8K ~0.64-0.68 at every checkpoint. Addresses review feedback on vllm-project#47206. Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
d82c7ee to
e8849fd
Compare
e8849fd to
81b090e
Compare
@SageMoore good question. These are single-node numbers, so it's not a multi-node fabric issue. Exact workload is the CI test itself (tests/distributed/test_elastic_ep.py::test_elastic_ep_scaling[sync_eplb]): DeepSeek-V2-Lite-Chat on one MI355X node, 4 GPUs, TP=1, DP=2, --enable-eplb sync mode, step_interval=10, window_size=5, num_redundant_experts=0. 26 MoE layers * 64 experts = 1664 physical expert slots. |
81b090e to
a93ea35
Compare
AndreasKaratzas
left a comment
There was a problem hiding this comment.
LGTM
@tlrmchlsmth PTAL
| _gl = global_expert_load_window | ||
| _ep = ep_group.size() | ||
|
|
||
| def _rank_imbalance( | ||
| _p2l: torch.Tensor, _gl: torch.Tensor, _ep: int | ||
| ) -> float: | ||
| _p2l = _p2l.to(_gl.device) | ||
| _pl = torch.gather(_gl, 1, _p2l.clamp(min=0).long()) | ||
| _pl = _pl * (_p2l >= 0) | ||
| _P = _p2l.shape[1] | ||
| _per_rank = ( | ||
| _pl.reshape(_pl.shape[0], _ep, _P // _ep) | ||
| .sum(dim=(0, 2)) | ||
| .float() | ||
| ) | ||
| _mean = _per_rank.mean() | ||
| if _mean <= 0: | ||
| return 1.0 | ||
| return (_per_rank.max() / _mean).item() | ||
|
|
||
| _old_imb = _rank_imbalance( | ||
| eplb_model_state.physical_to_logical_map, _gl, _ep | ||
| ) | ||
| _new_imb = _rank_imbalance(new_physical_to_logical_map, _gl, _ep) | ||
| local_skip = ( | ||
| _old_imb <= 1e-6 or (_old_imb - _new_imb) / _old_imb < 0.05 | ||
| ) | ||
| _vote = torch.tensor( | ||
| [1 if local_skip else 0], | ||
| device=_gl.device, | ||
| dtype=torch.int32, | ||
| ) | ||
| all_reduce(_vote, group=ep_group) | ||
| skip_rearrange = int(_vote.item()) == _ep |
There was a problem hiding this comment.
| _gl = global_expert_load_window | |
| _ep = ep_group.size() | |
| def _rank_imbalance( | |
| _p2l: torch.Tensor, _gl: torch.Tensor, _ep: int | |
| ) -> float: | |
| _p2l = _p2l.to(_gl.device) | |
| _pl = torch.gather(_gl, 1, _p2l.clamp(min=0).long()) | |
| _pl = _pl * (_p2l >= 0) | |
| _P = _p2l.shape[1] | |
| _per_rank = ( | |
| _pl.reshape(_pl.shape[0], _ep, _P // _ep) | |
| .sum(dim=(0, 2)) | |
| .float() | |
| ) | |
| _mean = _per_rank.mean() | |
| if _mean <= 0: | |
| return 1.0 | |
| return (_per_rank.max() / _mean).item() | |
| _old_imb = _rank_imbalance( | |
| eplb_model_state.physical_to_logical_map, _gl, _ep | |
| ) | |
| _new_imb = _rank_imbalance(new_physical_to_logical_map, _gl, _ep) | |
| local_skip = ( | |
| _old_imb <= 1e-6 or (_old_imb - _new_imb) / _old_imb < 0.05 | |
| ) | |
| _vote = torch.tensor( | |
| [1 if local_skip else 0], | |
| device=_gl.device, | |
| dtype=torch.int32, | |
| ) | |
| all_reduce(_vote, group=ep_group) | |
| skip_rearrange = int(_vote.item()) == _ep | |
| logical_loads = global_expert_load_window.float() | |
| ep_size = ep_group.size() | |
| def rank_load_imbalance(mapping: torch.Tensor) -> float: | |
| mapping = mapping.to( | |
| device=logical_loads.device, | |
| dtype=torch.long, | |
| ) | |
| replica_counts = torch.zeros_like(logical_loads) | |
| replica_counts.scatter_add_( | |
| dim=1, | |
| index=mapping, | |
| src=torch.ones_like(mapping, dtype=logical_loads.dtype), | |
| ) | |
| loads_per_replica = torch.gather( | |
| logical_loads / replica_counts.clamp_min(1), | |
| dim=1, | |
| index=mapping, | |
| ) | |
| loads_per_rank = loads_per_replica.reshape( | |
| logical_loads.shape[0], ep_size, -1 | |
| ).sum(dim=(0, 2)) | |
| mean_load = loads_per_rank.mean() | |
| if mean_load == 0: | |
| return 1.0 | |
| return (loads_per_rank.max() / mean_load).item() | |
| current_imbalance = rank_load_imbalance( | |
| eplb_model_state.physical_to_logical_map | |
| ) | |
| proposed_imbalance = rank_load_imbalance( | |
| new_physical_to_logical_map | |
| ) | |
| relative_improvement = ( | |
| current_imbalance - proposed_imbalance | |
| ) / current_imbalance | |
| skip_rearrange = relative_improvement < 0.05 |
There was a problem hiding this comment.
Trying to simplify + clean up this block of code before landing - @okorzh-amd and @AndreasKaratzas could you take a look?
There was a problem hiding this comment.
We should conservatively skip rearranging on non-AMD hardware as well IMO, since it adds an allreduce.
Adopt @tlrmchlsmth's suggestion for the low-gain rearrange skip check (PR vllm-project#47206, r3634076167): - rank_load_imbalance() now splits each logical expert's load across its physical replicas (scatter_add replica_counts, divide load) so the imbalance estimate matches how runtime traffic actually distributes. Behavior is identical when num_redundant_experts=0 (the test config) and strictly more accurate once experts are replicated. - Drop the per-rank all_reduce consensus vote: global_expert_load_window is already all-reduced and policy.rebalance_experts is deterministic, so every rank derives the same decision without an extra collective. - Remove now-dead guards (imbalance = max/mean is always >= 1.0) and the clamp/mask (the outer map >= 0 guard covers it). Gate the whole skip path behind current_platform.is_rocm(): the redundant- rearrange accuracy collapse this guards against was only observed on ROCm; other platforms keep the unconditional rearrange behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: okorzh <okorzh@amd.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: okorzh <okorzh@amd.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: okorzh <okorzh@amd.com>
558e33c to
908e774
Compare
|
Hi @okorzh-amd, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
rank_load_imbalance() is defined inside the per-model loop and closed over the loop-scoped logical_loads / ep_size, tripping ruff B023. Bind them as default arguments (evaluated at def time) so the closure captures the current iteration's values. Signed-off-by: okorzh <okorzh@amd.com>
|
Hi @okorzh-amd, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
Hi @okorzh-amd, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
…ject#47206) Signed-off-by: okorzh <okorzh@amd.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Each policy returns (changed, per_layer_priority, new_placement), and
policy_abstract documents the first value as 1 = table_changed, 0 =
not_changed. EplbWorker.do_update drops it:
_, _, new_placement = self.calculate_rebalance_experts(...)
so every plan is applied, including ones the policy itself judged not worth
applying. In DefaultEplb and SwiftBalanceEplb the flag is 1 only when the plan
cuts the summed per-layer peak load by at least 5%; a plan below that bar
still moves expert weights between NPUs for no gain.
This reads the flag and keeps the current placement when it is 0. That needs
no new control flow: every layer then compares equal in
compose_expert_update_info_greedy, which yields empty send and receive sets,
and the loader completes its cycle without moving anything. That path already
runs today for any layer whose map does not change. Returning early would not
be safe, because forward_before calls pop(0) on whatever the worker queues.
The decision is the same on every rank: load_info is the all-gathered load and
the policy and this check are deterministic, so no rank can diverge.
The same path also replaces the early return when no load is available. That
return queues None, which forward_before stores and the next pop(0) turns into
an AttributeError. Nothing reaches it today because warm_up_eplb always
gathers a load first, but keeping the placement removes the latent crash.
There is precedent upstream: vLLM core skips a rearrangement when the relative
improvement in imbalance is under 5% (vllm/distributed/eplb/eplb_state.py,
added in vllm-project/vllm#47206, currently ROCm only).
per_layer_priority is still discarded. It would matter if the number of layers
moved per cycle were ever capped; this leaves it alone.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Dylan Everingham <dceveringham@gmail.com>
Each policy returns (changed, per_layer_priority, new_placement), and
policy_abstract documents the first value as 1 = table_changed, 0 =
not_changed. EplbWorker.do_update drops it:
_, _, new_placement = self.calculate_rebalance_experts(...)
so every plan is applied, including ones the policy itself judged not worth
applying. In DefaultEplb and SwiftBalanceEplb the flag is 1 only when the plan
cuts the summed per-layer peak load by at least 5%; a plan below that bar
still moves expert weights between NPUs for no gain.
This reads the flag and keeps the current placement when it is 0. That needs
no new control flow: every layer then compares equal in
compose_expert_update_info_greedy, which yields empty send and receive sets,
and the loader completes its cycle without moving anything. That path already
runs today for any layer whose map does not change. Returning early would not
be safe, because forward_before calls pop(0) on whatever the worker queues.
The decision is the same on every rank: load_info is the all-gathered load and
the policy and this check are deterministic, so no rank can diverge.
The same path also replaces the early return when no load is available. That
return queues None, which forward_before stores and the next pop(0) turns into
an AttributeError. Nothing reaches it today because warm_up_eplb always
gathers a load first, but keeping the placement removes the latent crash.
There is precedent upstream: vLLM core skips a rearrangement when the relative
improvement in imbalance is under 5% (vllm/distributed/eplb/eplb_state.py,
added in vllm-project/vllm#47206, currently ROCm only).
per_layer_priority is still discarded. It would matter if the number of layers
moved per cycle were ever capped; this leaves it alone.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Dylan Everingham <dceveringham@gmail.com>
Purpose
Two fixes for distributed/test_elastic_ep.py, which collapsed GSM8K accuracy at the initial and scaled checkpoints on ROCm:
Bug 1 Scale-up produces garbage output on ROCm (accuracy=0)
Root cause. New elastic-EP ranks are created with load_dummy_weights=True. On ROCm, initialize_single_dummy_weight() .zero_()s every non-floating-point tensor in model.state_dict(). The FusedMoE expert-topology buffers expert_map, expert_mask, and expert{global_to_physical,physical_to_global,local_to_global} were registered as persistent integer buffers, so they were zeroed right after being built correctly for the new EP topology. That corrupted all2all dispatch garbage output. CUDA never zeroes integer tensors, so the bug was ROCm-only.
This corrects an earlier hypothesis: the map is not built unsharded at ep_size==1; it is built correctly at the target ep_size and then corrupted. (Thanks @itayalroy for identifying the dummy-init zeroing path.)
Fix. Register those five buffers with persistent=False. They are per-rank, deterministically-derived EP topology metadata not checkpoint state so they should never be in state_dict() (matches moondream3.py's _expert_map registration). This keeps them out of state_dict(), so:
As a result, no post-reshuffle metadata rebuild is needed and no name-based transfer filter for these tensors is needed.
Bug 2 Sync-EPLB accuracy loss from redundant rearranges
rearrange_expert_weights_inplace() was firing every step_interval even when the proposed map barely improved balance (imbalance 1.0), causing ~16s lateral all-to-all churn that starved the engine and dropped accuracy. We now compute per-rank load imbalance for the current vs proposed map and skip the rearrange when improvement is <5%, decided collectively via all-reduce and guarded to non-scale-up rearranges.
Test Plan
run failed CI tests
Test Result
Per-checkpoint accuracies (all >= 0.58 threshold; previously 0.19 / 0.016):
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.