Repository navigation
[NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe #40772
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
12b64f3
31b40c4
5760ba4
440d467
6c7edc7
ce108a5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1560,9 +1560,19 @@ def _pipeline_parallel_overlap_disable(view: Any) -> dict: | |
| def _speculative_moe_runner_default(view: Any) -> dict: | ||
| """Default the speculative (draft) MoE runner backend to the resolved | ||
| target-model backend. Invoked at the head of the speculative-decoding | ||
| hook, after the MoE kernel chain has resolved.""" | ||
| hook, after the MoE kernel chain has resolved. | ||
|
|
||
| megamoe's global-expert-id dispatch assumes a quantized layer; MTP/NextN | ||
| drafts never are, so inheriting it here mismatches their actual (Triton) | ||
| runner and corrupts expert-id dispatch (issue #40623; | ||
| _deepseek_spec_moe_resolution above carries the same fact for the | ||
| HIP+DeepSeek-fp4 case). Fall back to auto instead. | ||
| """ | ||
| if view.speculative_moe_runner_backend is None: | ||
| return {"speculative_moe_runner_backend": view.moe_runner_backend} | ||
| default = view.moe_runner_backend | ||
| if default == "flashinfer_megamoe": | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what about deepgemm megamoe?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DeepGEMM megamoe is selected via the a2a backend (megamoe), not through the global-expert-id dispatch path that flashinfer_megamoe uses. It also falls back per layer when its weights aren't prepared, so a draft can inherit it safely. I left it unchanged and added a comment saying so.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also, what happens if the draft model's moe is also quantized? Should we allow MegaMoE for those cases by default for draft?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can't tell at this point whether the draft is quantized, so the default is flashinfer_trtllm for the runner and none for the a2a backend. A quantized draft can opt in with --speculative-moe-runner-backend and --speculative-moe-a2a-backend. This is documented in the comments and help text. |
||
| default = "auto" | ||
|
wenscarl marked this conversation as resolved.
Resolved
|
||
| return {"speculative_moe_runner_backend": default} | ||
| return {} | ||
|
|
||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -440,6 +440,20 @@ def get_deepep_v2_fp8_scale_format() -> DeepEPv2Fp8ScaleFormat: | |
| ) | ||
|
|
||
|
|
||
| def _default_speculative_a2a_backend(target_a2a_backend: MoeA2ABackend) -> MoeA2ABackend: | ||
| if target_a2a_backend.is_flashinfer_megamoe(): | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what about deepgemm megamoe?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same above.
wenscarl marked this conversation as resolved.
Resolved
|
||
| return MoeA2ABackend.NONE | ||
| return target_a2a_backend | ||
|
|
||
|
|
||
| def _default_speculative_moe_runner_backend( | ||
| target_runner_backend: MoeRunnerBackendLike, | ||
| ) -> MoeRunnerBackendLike: | ||
| if target_runner_backend.is_flashinfer_megamoe(): | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same here: what about deepgemm megamoe and what if the draft moe is actually quantized?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same above. |
||
| return MoeRunnerBackend.FLASHINFER_TRTLLM | ||
| return target_runner_backend | ||
|
|
||
|
|
||
| def initialize_moe_config(): | ||
| """Seed the MoE runtime flags from the published configuration. | ||
|
|
||
|
|
@@ -457,12 +471,12 @@ def initialize_moe_config(): | |
| moe.speculative_runner_backend = ( | ||
| resolve_moe_runner_backend(spec.speculative_moe_runner_backend) | ||
| if spec.speculative_moe_runner_backend is not None | ||
| else moe.runner_backend | ||
| else _default_speculative_moe_runner_backend(moe.runner_backend) | ||
| ) | ||
| moe.speculative_a2a_backend = ( | ||
| MoeA2ABackend(spec.speculative_moe_a2a_backend) | ||
| if spec.speculative_moe_a2a_backend is not None | ||
| else moe.a2a_backend | ||
| else _default_speculative_a2a_backend(moe.a2a_backend) | ||
| ) | ||
| moe.deepep_mode = DeepEPMode(exec_moe.deepep_mode) | ||
| moe.deepep_config = exec_moe.deepep_config or "" | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: modify the comment so that they are not so specific to issue 40623. Make it more general.
For example:
"drafts never are" is too strong. Some MTPs are indeed quantized.
"actual (Triton)" is too strong. Triton is not always the actual runner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reworded. It no longer cites the issue, and it says drafts are 'typically' unquantized. It also no longer names Triton as the runner.