Skip to content
Closed
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
8af1a00
Fix NaN gradients in expert-parallel training: mask uninitialized gro…
qgallouedec Aug 21, 2026
1f0b7bf
Fix wrong gradients for all non-expert parameters in expert-parallel …
qgallouedec Aug 21, 2026
4636d42
Trim comments
qgallouedec Aug 22, 2026
89fa55c
Gate the router-score backward all-reduce on grad mode; drop the post…
qgallouedec Sep 3, 2026
96769d9
Fix expert parallelism through Trainer (#48208)
qgallouedec Sep 3, 2026
c74c122
Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh
qgallouedec Sep 3, 2026
1c66acb
Gather the sharded state dict through the model's own save helpers
qgallouedec Sep 3, 2026
38aefbf
Document combining expert parallelism with FSDP2
qgallouedec Sep 3, 2026
e72b117
Address review: drop the dead expert-gradient sync, torch-native mixe…
qgallouedec Sep 3, 2026
449acd2
Give each device mesh its own optimizer param group instead of steppi…
qgallouedec Sep 3, 2026
9d17bd7
Test the 2-D mesh with a batch split across the fsdp ranks and compar…
qgallouedec Sep 3, 2026
6f2d9c4
Refresh the throughput table
qgallouedec Sep 3, 2026
3a8da11
Cosmetic: docstring layout, comment placement, doc link
qgallouedec Sep 3, 2026
1b07199
Expert-parallel token dispatch: each rank trains on its own part of t…
qgallouedec Sep 4, 2026
7063c45
Make the dispatch handling of the experts explicit in apply_fully_sha…
qgallouedec Sep 4, 2026
dc1fdee
Combine the dispatched expert outputs in the routing weights' precisi…
qgallouedec Sep 4, 2026
f2a3c26
Split the batches across every rank with a DistributedSampler under t…
qgallouedec Sep 4, 2026
3ab94ff
Simplify the dispatch: torch's differentiable all-to-all, the grouped…
qgallouedec Sep 4, 2026
d188144
Dispatch as plan-level styles; one notion of the ranks sharing a batc…
qgallouedec Sep 4, 2026
b9bf58b
Select the dispatch styles in the mixin, guard the torch version and …
qgallouedec Sep 4, 2026
39cdce5
Rewrite the expert parallel plan the model actually reads for the dis…
qgallouedec Sep 4, 2026
9f86230
Seed the dispatch sampler with data_seed; require the grouped_mm expe…
qgallouedec Sep 4, 2026
930c146
Run the local experts directly: the grouped forward's masking and com…
qgallouedec Sep 4, 2026
15588e0
Run the local experts through the module's own forward
qgallouedec Sep 4, 2026
51f281d
Dispatch: one host sync per layer, comments say what the code does
qgallouedec Sep 4, 2026
101bfea
Dispatch: drop the router style, reuse the named mesh and is_dtensor,…
qgallouedec Sep 4, 2026
2d9046b
Dispatch: skip the mixin tests for models without an FSDP plan, drop …
qgallouedec Sep 4, 2026
42606af
Fix the _has_fsdp_plan docstring
qgallouedec Sep 4, 2026
d8eb1fc
Drop the FSDP plan skip from the mixin tests: every config carries a …
qgallouedec Sep 4, 2026
23920e3
Format
qgallouedec Sep 4, 2026
e149869
Keep the base install_forward signature on the dispatch experts style
qgallouedec Sep 4, 2026
e139e8a
Docs: shorter token dispatch overview
qgallouedec Sep 4, 2026
cf74929
Replicate the rest of the plan under expert parallel token dispatch
qgallouedec Sep 7, 2026
0e21b59
Merge remote-tracking branch 'upstream/main' into fsdp2-ep-2d-mesh
qgallouedec Sep 8, 2026
18144ab
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
qgallouedec Sep 8, 2026
c9a4216
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 8, 2026
8f2f9cb
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 8, 2026
3c8f600
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 8, 2026
e7be30a
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 8, 2026
8dbccad
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 8, 2026
35ed430
Merge branch 'main' into fsdp2-ep-2d-mesh
qgallouedec Sep 8, 2026
79fe11d
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
qgallouedec Sep 8, 2026
0817429
Keep replicated_with_grad_allreduce under expert-parallel dispatch
qgallouedec Sep 8, 2026
34e63c9
Compare gradients, not only the loss, in the EP backward test
qgallouedec Sep 9, 2026
75fff71
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 10, 2026
acba6bc
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 10, 2026
69668c3
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 10, 2026
9cfb2d3
Update docs/source/en/expert_parallelism.md
qgallouedec Sep 10, 2026
2884e2f
Merge branch 'main' into fsdp2-ep-2d-mesh
qgallouedec Sep 10, 2026
4ce4700
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
qgallouedec Sep 10, 2026
28cb482
Declare dispatch_experts_forward next to its only caller
qgallouedec Sep 10, 2026
a6e36c0
Name the dispatch strategy instead of switching it with a bool
qgallouedec Sep 10, 2026
1be3a7f
Merge branch 'main' into fsdp2-ep-2d-mesh
qgallouedec Sep 10, 2026
db78860
Merge remote-tracking branch 'upstream/fsdp2-ep-2d-mesh' into ep-toke…
qgallouedec Sep 10, 2026
a6f5824
Merge branch 'main' into fsdp2-ep-2d-mesh
qgallouedec Sep 10, 2026
b56e1d6
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
qgallouedec Sep 10, 2026
cb02b27
Merge branch 'main' into fsdp2-ep-2d-mesh
qgallouedec Sep 11, 2026
6a86e18
Pass the ignored params through the policy helper
qgallouedec Sep 11, 2026
79988c3
Merge remote-tracking branch 'upstream/main' into fsdp2-ep-2d-mesh
qgallouedec Sep 12, 2026
5120748
Drop the comment on the parallel sizes
qgallouedec Sep 12, 2026
0147840
Trim the parallelism config comment
qgallouedec Sep 12, 2026
d604a2a
Merge remote-tracking branch 'upstream/fsdp2-ep-2d-mesh' into ep-toke…
qgallouedec Sep 12, 2026
b169a75
Merge remote-tracking branch 'upstream/main' into ep-token-dispatch
qgallouedec Sep 12, 2026
3439765
Address review: guard dispatch + PP, derive the mesh name, move the g…
qgallouedec Sep 13, 2026
4f2a6b9
Keep reading the tensor parallel size from accelerate in `get_tp_size`
qgallouedec Sep 13, 2026
86561b6
Give the token dispatch weight comparison its own tolerance
qgallouedec Sep 13, 2026
56047f4
Merge branch 'main' into ep-token-dispatch
3outeille Sep 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions docs/source/en/expert_parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,50 @@ Launch your inference script with [torchrun](https://pytorch.org/docs/stable/ela
torchrun --nproc-per-node 8 your_script.py
```

## Token dispatch

By default every expert-parallel rank runs the whole batch, keeps only the experts it owns, and the group all-reduces the expert outputs at the end of every MoE layer. The parameters outside the experts do `tp_size` times the same work, and the all-reduce moves the full activations. `expert_parallel_dispatch=True` sends each token to the rank that owns its experts instead: this means a lot less data needs to travel between GPUs / nodes in large scale trainings.
Comment thread
qgallouedec marked this conversation as resolved.
Resolved

```py
distributed_config = DistributedConfig(
Comment thread
qgallouedec marked this conversation as resolved.
Resolved
tp_size=8,
enable_expert_parallel=True,
expert_parallel_dispatch=True,
)
```

Each rank trains on its own part of the batch. At every MoE layer it routes its tokens, sends each (token, expert) pair to the rank that owns the expert with an all-to-all, runs its local experts, and gets the results back with a second all-to-all. Only the routed tokens travel.

The rest of the model is handled as follows:
Comment thread
qgallouedec marked this conversation as resolved.
Resolved

- The parameters outside the experts are data-parallel across the whole group, so they are sharded with [FSDP2](./fsdp) across every rank (`fsdp` and `tp` together when both are set), and FSDP2 reduces their gradients.
- The experts stay sharded across `tp`, and across `fsdp` too when `fsdp_size > 1`. With `fsdp_size=1` they are outside FSDP2, so `fsdp_mixed_precision` and `fsdp_cpu_offload` do not apply to them.
- The [`Trainer`] gives each rank its own training batches and counts tokens across all of them. Evaluation is unchanged from plain expert parallelism: every `tp` rank sees the same batches.

Comment thread
qgallouedec marked this conversation as resolved.
Resolved
## Combining with FSDP2

Expert parallelism only shards the experts. Everything else (attention, embeddings, norms) and its optimizer state is replicated on every expert-parallel rank, which is what limits the model size you can train. Set `fsdp_size` together with `tp_size` to add [FSDP2](./fsdp) on a second mesh dimension.

```py
distributed_config = DistributedConfig(
tp_size=4, # expert parallel size
fsdp_size=2, # data parallel shards
enable_expert_parallel=True,
)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-30B-A3B", distributed_config=distributed_config)
```

The model is loaded on a 2-D `(fsdp, tp)` device mesh, and `tp_size * fsdp_size` must equal the number of processes. The expert parallel plan shards the experts across `tp`, then FSDP2 shards every parameter, experts included, across `fsdp` and owns their gradient reduction. Each `fsdp` rank trains on its own part of the batch. Nothing else changes: train with the [`Trainer`] as usual (it computes the gradient norm across the two meshes and gives each mesh its own optimizer param group), and [`~Trainer.save_model`] gathers the sharded weights and writes a regular checkpoint.

On 8 GPUs, full fine-tuning of Qwen3-30B-A3B in bf16 at sequence length 2048:

| configuration | tokens/s/GPU | peak memory/GPU |
|---|---|---|
| `tp_size=8` | 3485 | 38.6 GB |
| `tp_size=4, fsdp_size=2` | 2900 | 34.2 GB |
| `tp_size=2, fsdp_size=4` | 2830 | 32.3 GB |

> [!WARNING]
> Resuming from a checkpoint is not supported yet for models sharded at load time, so the [`Trainer`] only accepts `save_only_model=True` or `save_strategy="no"` for them.

[[autodoc]] DistributedConfig
3 changes: 3 additions & 0 deletions docs/source/en/fsdp.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,9 @@ TrainingArguments(
</hfoption>
</hfoptions>

> [!TIP]
> For mixture-of-experts models, `fsdp_size` can be combined with `tp_size` and `enable_expert_parallel=True` to shard the experts across one mesh dimension and everything else across the other. See [expert parallelism](./expert_parallelism#combining-with-fsdp2).
## Next steps

- See [DDP](./ddp) for data-parallel training when your model fits on one GPU.
Expand Down
15 changes: 11 additions & 4 deletions src/transformers/distributed/configuration_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ class DistributedConfig:
Reserved for sequence parallelism. Not wired up yet.
enable_expert_parallel (`bool`, *optional*, defaults to `False`):
Route MoE models through the expert-parallel path (``base_model_ep_plan``).
expert_parallel_dispatch (`bool`, *optional*, defaults to `False`):
Send each token to the rank that owns its experts with an all-to-all. Each rank trains on its own part of
the batch, and the parameters that are not expert-parallel are sharded with FSDP2 across every rank.
Requires `enable_expert_parallel`.
fsdp_size (`int`, *optional*):
Number of devices for FSDP (data parallelism). If `None` and `tp_size` is set, defaults to 1.
fsdp_cpu_offload (`bool`, *optional*, defaults to `False`):
Expand All @@ -48,6 +52,7 @@ class DistributedConfig:
tp_plan: dict[str, str] | Literal["auto"] | None = None
enable_sequence_parallel: bool = False
enable_expert_parallel: bool = False
expert_parallel_dispatch: bool = False
fsdp_size: int | None = None
fsdp_cpu_offload: bool = False
fsdp_mixed_precision: bool = False
Expand All @@ -73,11 +78,13 @@ def __post_init__(self):
elif self.tp_size is None:
self.tp_size = 1

if self.tp_size > 1 and self.fsdp_size > 1 and self.pp_size > 1:
if self.expert_parallel_dispatch and not self.enable_expert_parallel:
raise ValueError("`expert_parallel_dispatch=True` requires `enable_expert_parallel=True`.")

if self.pp_size > 1 and (self.tp_size > 1 or self.fsdp_size > 1):
raise ValueError(
"FSDP+TP+PP is not supported yet. "
"Use DistributedConfig(fsdp_size=N) or DistributedConfig(tp_size=N) or DistributedConfig(pp_size=N), not all three. "
"Only 1D support is available for now."
"Pipeline parallelism cannot be combined with tensor or FSDP parallelism yet. "
"Use DistributedConfig(pp_size=N) on its own, or DistributedConfig(tp_size=N, fsdp_size=M)."
)

@classmethod
Expand Down
24 changes: 22 additions & 2 deletions src/transformers/distributed/fsdp.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from ..utils import is_torch_available, is_torch_distributed_available, is_torch_greater_or_equal, logging, strtobool
from ..utils.quantization_config import QuantizationMethod
from .tensor_parallel import replace_layer_number_by_wildcard
from .utils import _is_torch_distributed_initialized
from .utils import _is_torch_distributed_initialized, is_dtensor


if TYPE_CHECKING:
Expand Down Expand Up @@ -185,13 +185,19 @@ def verify_fsdp_plan(module_names: list[str], fsdp_plan: dict[str, str] | None)


def apply_fully_sharded_data_parallelism(
model: nn.Module, fsdp_mesh: torch.distributed.device_mesh.DeviceMesh
model: nn.Module,
fsdp_mesh: torch.distributed.device_mesh.DeviceMesh,
expert_mesh: torch.distributed.device_mesh.DeviceMesh | None = None,
) -> nn.Module:
"""
Apply FSDP2 (fully_shard) to a model.

Torch availability, distributed initialization and the version requirement
are asserted upstream by `initialize_fully_sharded_data_parallelism`.

With expert-parallel token dispatch `fsdp_mesh` spans the expert-parallel ranks, which the experts are already
sharded across: the experts are fully sharded across `expert_mesh` in their own group, and passed to FSDP2 as
`ignored_params` when `expert_mesh` is `None`.
"""
fsdp_plan = dict(getattr(model, "_fsdp_plan", None) or {})
if not fsdp_plan:
Expand All @@ -206,6 +212,20 @@ def apply_fully_sharded_data_parallelism(
adapted_fsdp_plan = _resolve_tied_embed_lm_head_plan(fsdp_plan, model)
reshard_targets, no_reshard_targets = expand_fsdp_plan(model, adapted_fsdp_plan)

if distributed_config is not None and distributed_config.expert_parallel_dispatch:
if not is_torch_greater_or_equal("2.7"):
raise OSError("Expert-parallel token dispatch requires `torch>=2.7`.")
# The DTensor parameters are the expert-parallel experts: `maybe_distribute_model` rewrote the expert
# parallel plan to shard only them.
expert_modules = [
module for module in model.modules() if any(is_dtensor(p) for p in module.parameters(recurse=False))
Comment thread
qgallouedec marked this conversation as resolved.
Resolved
]
if expert_mesh is not None:
for module in expert_modules:
fully_shard(module, mesh=expert_mesh, reshard_after_forward=True, **fsdp_policy_kwargs)
else:
fsdp_policy_kwargs["ignored_params"] = {p for module in expert_modules for p in module.parameters()}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a small update but:

    fsdp_plan = dict(getattr(model, "_fsdp_plan", None) or {})

setting a custom plan will make sure onlyl these are used

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should of course be in the distributed config

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the DTensor sniff is fragile: it happens to be exact only because the block above rewrites _ep_plan to expert-only styles, so it depends on something 40 lines away.
_fsdp_plan is already resolved at line 202 in this function, so it's the natural handle.

You said you'd take this one, so I'll leave it to you 👍

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain a bit why we need to ignored_params here ? It is not obvious to me

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the fsdp_size == 1, tp_size > 1 case: the mesh is 1-D, so there is no fsdp dimension to put the experts on.

Under dispatch the trunk is DP over the whole mesh, so fully_shard runs on the trunk modules with that mesh. The experts are not DP: each rank owns a different slice of them and they are already sharded across tp.

Without ignoring them they land in those param groups, and FSDP2 tries to concatenate its mesh with the parameter's own one and dies at load. Nothing transformers-specific:

# torchrun --nproc_per_node 2 repro.py
import torch, torch.distributed as dist
from torch.distributed.fsdp import fully_shard
from torch.distributed.tensor import Shard, distribute_tensor

dist.init_process_group("gloo")
mesh = torch.distributed.init_device_mesh("cpu", (dist.get_world_size(),))

class Block(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.trunk = torch.nn.Linear(8, 8, bias=False)
        self.experts = torch.nn.Linear(8, 8, bias=False)

block = Block()
block.experts.weight = torch.nn.Parameter(distribute_tensor(block.experts.weight.data, mesh, [Shard(0)]))
fully_shard(block, mesh=mesh)
_fsdp_param.py:501, in _init_sharding_spec_tp
device_mesh.py:1437, in _concatenate
TypeError: Invariant encountered: value was None when it should not be

The mesh initialize_tensor_parallelism builds has no mesh_dim_names, which is the None.

(The branch just above is the other case: when fsdp_size > 1 there is an fsdp dimension, so the experts get their own fully_shard on expert_mesh instead of being ignored.)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small correction to the above: since #48155/#48516 the mesh is always named, so the failure is now RuntimeError: Cannot concatenate overlapping meshes rather than the None invariant. The reason is the same, FSDP2's mesh and the experts' mesh are the same ranks.

Comment thread
qgallouedec marked this conversation as resolved.
Resolved

for module_name, module in reshard_targets:
fully_shard(module, mesh=fsdp_mesh, reshard_after_forward=True, **fsdp_policy_kwargs)
logger.debug(f"Applied fully_shard to {module_name} (reshard=True)")
Expand Down
71 changes: 57 additions & 14 deletions src/transformers/distributed/mixin.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ class DistributedMixin:
_tp_plan: dict[str, str] | None = None
_ep_plan: dict[str, str] | None = None
_tp_size = None
_fsdp_size = None
_expert_parallel_dispatch = False
_pp_plan: dict[str, tuple[str, str]] | None = None
_fsdp_plan: dict[str, str] | None = None

Expand Down Expand Up @@ -163,17 +165,23 @@ def prepare_distribute_model(
f"is not equal to world_size ({world_size})"
)

if distributed_config.tp_size > 1:
if distributed_config.tp_plan is None:
distributed_config.tp_plan = "auto"
if distributed_config.tp_size > 1 and distributed_config.tp_plan is None:
distributed_config.tp_plan = "auto"

if distributed_config.fsdp_size > 1:
# Builds a 2-D (fsdp, tp) mesh when tensor/expert parallelism is also requested.
if device_mesh is not None:
raise ValueError(
"`device_mesh` cannot be passed together with `fsdp_size > 1`: the mesh is built here."
)
device_map, device_mesh = initialize_fully_sharded_data_parallelism(distributed_config)
elif distributed_config.tp_size > 1:
device_map, device_mesh = initialize_tensor_parallelism(
distributed_config.tp_plan,
tp_size=distributed_config.tp_size,
device_mesh=device_mesh,
device_map=device_map,
)
elif distributed_config.fsdp_size > 1:
device_map, device_mesh = initialize_fully_sharded_data_parallelism(distributed_config)
elif distributed_config.pp_size > 1:
device_map, device_mesh = initialize_pipeline_parallelism(distributed_config)

Expand All @@ -190,19 +198,52 @@ def maybe_distribute_model(
if device_mesh is not None:
model.config.distributed_config = distributed_config
model._device_mesh = device_mesh
# The Trainer mirrors these into accelerate's `ParallelismConfig`; without them accelerate
# sees unaccounted ranks and falls back to DDP, which rejects the DTensor parameters.
model._tp_size = distributed_config.tp_size
model._fsdp_size = distributed_config.fsdp_size
model._expert_parallel_dispatch = distributed_config.expert_parallel_dispatch

# Both may apply: the tensor/expert parallel plan shards across `tp` first, then FSDP2
# shards every parameter (the `tp`-sharded ones included) across `fsdp`.
if distributed_config.tp_size > 1:
tp_mesh = device_mesh["tp"] if device_mesh.ndim > 1 else device_mesh
if isinstance(distributed_config.tp_plan, dict):
model.tp_plan = distributed_config.tp_plan
if distributed_config.expert_parallel_dispatch:
# Every rank trains on its own part of the batch, so only the experts can be sharded across the
# group: the experts get the dispatch style, the router keeps its global ids and scores, and
# whatever else the plan shards stays replicated, data-parallel like the rest of the trunk.
replicated = sorted(
name
for name, style in model.tp_plan.items()
if style not in ("ep_router", "grouped_gemm", "moe_tp_experts")
)
Comment on lines +199 to +201

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty sure default is replicate, so if you pass the ep plan hard coded, it won't follow the one form the config. You don't need any of that in that case no?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not setting a default, it's dropping entries.
tp_plan reads _ep_plan under EP, and dispatch can only shard the experts, so the rewrite keeps grouped_gemm/moe_tp_experts and drops everything else the plan would otherwise shard.
Without it a plan that shards e.g. attention would still shard it, and each rank is training on its own tokens, so that would be wrong rather than just wasteful.
The warning lists whatever got dropped.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's the wrong place to hardcode the style names though. If it moves behind _fsdp_plan in the distributed config as you suggested, this block should go with it. 👌

if replicated:
logger.warning(
"`expert_parallel_dispatch=True` shards only the experts, so these expert parallel plan "
f"entries are ignored and their modules stay replicated: {replicated}."
)
# `tp_plan` reads `_ep_plan` under expert parallelism, so that is the plan to rewrite.
model._ep_plan = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to do the dispatch of plan at the same place where tp_plan reads _ep_plan (https://github.com/huggingface/transformers/blob/ep-token-dispatch/src/transformers/distributed/mixin.py#L89) wdyt ?

at this stage of the code, we should only be reading & applying the plan. Not modifying it

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you do that, you can probably check for replicate by just doing set(model._ep_plan) - set(model.tp_plan)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it should not be modifying the plan here. @ArthurZucker made the same point in #48518 (comment) and said he would take it, so I left it alone. Happy to move it into the property as you describe if he would rather not.

set(model._ep_plan) - set(model.tp_plan) works once the filtering is in there

(a property is read more than once, so it would need warning_once.)

name: "ep_dispatch_experts" if style == "moe_tp_experts" else style
for name, style in model.tp_plan.items()
if style in ("grouped_gemm", "moe_tp_experts")
}
model = apply_tensor_parallelism(model, tp_mesh)

if distributed_config.expert_parallel_dispatch:
# Every expert-parallel rank trains on its own part of the batch, so the parameters outside the
# experts are data-parallel across the whole mesh: FSDP2 shards them across all of it and owns
# their gradient reduction. The experts stay sharded across `tp` and, if any, across `fsdp`.
trunk_mesh = device_mesh["fsdp_tp"] if device_mesh.ndim > 1 else device_mesh
expert_mesh = device_mesh["fsdp"] if device_mesh.ndim > 1 else None
model = apply_fully_sharded_data_parallelism(model, trunk_mesh, expert_mesh=expert_mesh)
elif distributed_config.fsdp_size > 1:
fsdp_mesh = device_mesh["fsdp"] if device_mesh.ndim > 1 else device_mesh
model = apply_fully_sharded_data_parallelism(model, fsdp_mesh)

elif distributed_config.pp_size > 1:
if distributed_config.pp_size > 1:
pp_mesh = device_mesh["pp"] if device_mesh.ndim > 1 else device_mesh
model = apply_pipeline_parallelism(model, pp_mesh)
return model
Expand Down Expand Up @@ -265,6 +306,16 @@ def gather_sharded_state_dict_for_save(
if distributed_config is None:
return state_dict

if distributed_config.fsdp_size > 1 or distributed_config.expert_parallel_dispatch:
# Also covers the 2-D (fsdp, tp) mesh and token dispatch: every parameter is FSDP-managed, and
# the full state dict is only materialized on rank 0.
if not _is_torch_distributed_initialized():
raise ValueError(
"Saving an FSDP-wrapped model requires torch.distributed to be initialized. "
"Call save_pretrained from every rank after init_process_group."
)
return gather_full_state_dict(model_to_save)

if distributed_config.tp_size > 1:
state_dict = gather_state_dict_for_save(
state_dict, self._tp_plan, self._device_mesh, distributed_config.tp_size
Expand All @@ -273,14 +324,6 @@ def gather_sharded_state_dict_for_save(
state_dict = {}
return state_dict

if distributed_config.fsdp_size > 1:
if not _is_torch_distributed_initialized():
raise ValueError(
"Saving an FSDP-wrapped model requires torch.distributed to be initialized. "
"Call save_pretrained from every rank after init_process_group."
)
return gather_full_state_dict(model_to_save)

return state_dict

def barrier_after_gathered_checkpoint_save(self, distributed_config: DistributedConfig | None) -> None:
Expand Down
23 changes: 23 additions & 0 deletions src/transformers/distributed/tensor_parallel.py
Original file line number Diff line number Diff line change
Expand Up @@ -737,6 +737,28 @@ def transform_output_post_forward(self, module, output, mesh):
return output


class EpDispatchExpertsParallel(MoeExpertsParallel):
Comment thread
qgallouedec marked this conversation as resolved.
Resolved
"""Experts of expert-parallel token dispatch: every rank sends its own tokens to the experts' owners."""

def install_forward(self, module, mesh, *, is_expert_parallel=False):
from ..integrations.moe import dispatch_experts_forward

original_forward = module.forward
ep_group, ep_size = mesh.get_group(), mesh.size()

def tp_forward(hidden_states, top_k_index, top_k_weights):
if isinstance(hidden_states, DTensor):
hidden_states = hidden_states.to_local()
with self.context_around_forward(module, mesh):
# The sharding leaves the module with its local expert count.
return dispatch_experts_forward(
Comment thread
qgallouedec marked this conversation as resolved.
Resolved
original_forward, module.num_experts, hidden_states, top_k_index, top_k_weights, ep_group, ep_size
)

module.forward = tp_forward
return module


class MoeTensorParalellMegaMoeExperts(MoeExpertsParallel):
"""TP layer for DeepGEMM Mega MoE experts.

Expand Down Expand Up @@ -774,6 +796,7 @@ class ParallelInterface(GeneralInterface):
"sequence_parallel": SequenceParallel(use_local_output=True),
"grouped_gemm": MoEParamShard(Shard(0), shards_expert_dim=True),
"ep_router": EpRouterParallel(),
"ep_dispatch_experts": EpDispatchExpertsParallel(),
Comment thread
qgallouedec marked this conversation as resolved.
Resolved
"megamoe_router": RouterParallelMegaMoe(),
"moe_tp_experts": MoeExpertsParallel(),
"megamoe_experts": MoeTensorParalellMegaMoeExperts(),
Expand Down
6 changes: 6 additions & 0 deletions src/transformers/distributed/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -193,11 +193,17 @@ def initialize_fully_sharded_data_parallelism(distributed_config: DistributedCon
device_map = torch.device(device_type)

fsdp_size = distributed_config.fsdp_size
tp_size = distributed_config.tp_size

# `fsdp` is the outer dimension so that the `tp` ranks of a group are contiguous, which is what
# the expert all-to-all and the TP collectives want.
dims, names = [], []
if fsdp_size > 1:
dims.append(fsdp_size)
names.append("fsdp")
if tp_size > 1:
dims.append(tp_size)
names.append("tp")

# Build the N-dimensional device mesh
mesh = torch.distributed.init_device_mesh(device_type, tuple(dims), mesh_dim_names=tuple(names))
Expand Down
Loading
Loading