Repository navigation
Restore GPT-OSS's router-masked expert-parallel plan - #49431
IlyasMoutawwakil wants to merge 2 commits into
Conversation
The MXFP4 experts route their tokens themselves: they take the routing data `routing_torch_dist` builds (already restricted to the rank's experts) instead of expert ids, which token dispatch cannot exchange. With the plan on `ep_dispatch_experts`, MXFP4 GPT-OSS under expert parallelism failed on its first forward (`ep_forward() got an unexpected keyword argument 'scatter_idx'`). The router-masked plan from before #49160 leaves them to their own routing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
It inherits GPT-OSS's config through its modular file, but its experts take expert ids like every other experts class, so it keeps the `ep_dispatch_experts` plan rather than following GPT-OSS back to router masking. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
[For maintainers] Suggested jobs to run (before merge) run-slow: gpt_oss, openai_privacy_filter |
CI recapDashboard: View test results in Grafana |
MXFP4 GPT-OSS experts take routing data instead of expert ids; GPT-OSS keeps the router-masked plan for them (#49431), so masked EP only handles expert ids, and token dispatch points such experts to that plan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
| "layers": (["hidden_states", "attention_mask"], ["hidden_states"]), | ||
| "norm": (["hidden_states"], ["hidden_states"]), | ||
| } | ||
| # MXFP4 experts route their tokens themselves, which only the router-masked plan leaves them to |
There was a problem hiding this comment.
mmm this should not be related. MXFP4 should be acting on the plan AFAIK
There was a problem hiding this comment.
in an ideal world yes, but the current mxfp4 integration is based on kernels that aren't modular enough by default to do that. i can adapt this mxfp4 (which we will get rid of once we get to finegrained kernels) but it has to import some non-surfaced ops in the gpt oss mxfp4 kernels which turns iut to be much uglier.
There was a problem hiding this comment.
that's why i said this is temporary, until fg kernels is merged which does not have the limitations and rigidity of "triton-kernels", lmk what u prefer
|
closing in favor of the EP strategies stack: GPT-OSS (MXFP4 included) now runs on |
The MXFP4 experts route their tokens themselves: they take the routing data
routing_torch_distbuilds (already restricted to the rank's experts) instead of expert ids, which token dispatch cannot exchange. With the plan onep_dispatch_experts, MXFP4 GPT-OSS under expert parallelism failed on its first forward (ep_forward() got an unexpected keyword argument 'scatter_idx'). The router-masked plan from before #49160 leaves them to their own routing.What does this PR do?
Fixes # (issue)
Code Agent Policy
The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. These often are low-quality, or fix extremely minor issues that occur rarely or never in practice.
As a result, we're instituting a rule that first-time contributors should not use code agents to submit PRs or issues.
We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues.
Issues/PRs from first-time contributors that violate this rule will probably just be closed without review, and we
might block you, especially if you open more than one or appear to be deliberately ignoring this. We especially do not
want new contributors to jump in on random issues to contribute an agent-written fix. This creates lots of noise
for reviewers and other users and will almost certainly get you blocked.
For more information, please read
CONTRIBUTING.md.Before submitting
Pull Request checks?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.