You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 04dc993
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/en/expert_parallelism.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,7 +19,7 @@ rendered properly in your Markdown viewer.
19
19
20
20
## DistributedConfig
21
21
22
-
Enable expert parallelism with the [`DistributedConfig`] class and the `ep_size` argument. Most models route with masking and all-reduce, which requires `ep_size=tp_size` so every rank in an expert group receives the same tokens. Models whose plan uses [token dispatch](#token-dispatch), such as Qwen3 MoE, can set `ep_size` independently of `tp_size`.
22
+
Enable expert parallelism with the [`DistributedConfig`] class and the `ep_size` argument. Most MoE models default to [token dispatch](#token-dispatch), so `ep_size` can be set independently of `tp_size`. A few, such as Llama 4 and Gemma 4, still default to masking and all-reduce (`"ep_router"` and `"moe_tp_experts"` in `model.ep_plan`). Masking is also available on any model with an `ep_plan` override, and requires `ep_size=tp_size` so every rank in an expert group receives the same tokens.
Providing a plan does not infer parallel sizes: set `tp_size` and `ep_size` explicitly.
76
76
77
-
Qwen3 MoE defaults to `"ep_dispatch_experts"`. To use masking and all-reduce instead, set `ep_size=tp_size` and override both the router and the expert forward rules:
77
+
Most MoE models default to `"ep_dispatch_experts"`. To use masking and all-reduce instead, set `ep_size=tp_size` and override both the router and the expert forward rules (the router module name depends on the model, e.g. `mlp.router` on gpt-oss):
78
78
79
79
```py
80
80
distributed_config = DistributedConfig(
@@ -155,7 +155,7 @@ These configurations each use eight GPUs:
155
155
156
156
| Configuration | Result |
157
157
| :--- | :--- |
158
-
|`DistributedConfig(tp_size=4, fsdp_size=2, ep_size=4)`| Dispatch with TP groups of four, each slicing its batch in four (Qwen3 MoE default plan); unless you specify a ep_plan to use the legacy masked EP |
158
+
|`DistributedConfig(tp_size=4, fsdp_size=2, ep_size=4)`| Dispatch with TP groups of four, each slicing its batch in four (default plan); unless you specify a ep_plan to use the legacy masked EP |
159
159
|`DistributedConfig(tp_size=1, fsdp_size=8, ep_size=4)`| Dispatch with an independent batch on each rank, no slicing, and experts FSDP-sharded across pairs of ranks. |
160
160
|`DistributedConfig(tp_size=2, fsdp_size=4, ep_size=4)`| Dispatch with a TP pair per batch, each pair slicing its batch in two; two batches per expert group. |
161
161
|`DistributedConfig(tp_size=8, ep_size=8)`| Dispatch with every rank sharing one batch, sliced in eight, or masking and all-reduce for a masked plan. |
0 commit comments