Skip to content

Fix disagg PP MTP for GLM5.2 - #39052

Closed
nvjullin wants to merge 2 commits into
sgl-project:mainfrom
nvjullin:pp-hetero-mtp
Closed

nvjullin wants to merge 2 commits into
sgl-project:mainfrom
nvjullin:pp-hetero-mtp

Conversation

@nvjullin

@nvjullin nvjullin commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

This PR is based on #38853, which fixes disagg PP for MLA. This PR fixes disagg PP + MTP for GLM5.2.

There are hard errors when running PP prefill + DP decode on GLM5.2, caused by three things:

  1. PP + speculative decoding is rejected outright on every non-NPU platform, by check_server_args in arg_groups/validation_hook.py and by _assert_pp_mtp_compat in model_executor/model_runner_components/layer_setup.py. The runtime underneath is already PP-aware and platform-neutral: EAGLEWorkerV2 builds the draft on the last PP stage only (_hosts_draft) and every earlier stage forwards the target's PP proxy tensors and returns.
  2. Token embeddings are only available on first PP rank, but draft model also need token embeddings
  3. PP scheduler_pp_mixin drops the draft model's DSA IndexShare seed (dsa_topk_indices), so decode has to regenerate them. However, decode DP currently hangs if one rank receives nothing which will run cudagraph, and another rank receives a request with no topk indices which runs eagerly.

On (3): the last PP stage's draft proposal crosses the pipeline ring as draft_topk_p / draft_topk_index / draft_hidden_states (_pp_prepare_tensor_dict) and is rebuilt into a fresh EagleDraftInput on every rank, which is then rebound to batch.spec_info.

Modifications

  1. Allow PP + speculative decoding. validation_hook keeps the overlap-schedule and EAGLE-shape restrictions for all platforms and keeps NPU's extra prefill-only rule, but the platform branch is gone. Removed _assert_pp_mtp_compat and spec_algorithm argument to resolve_layer_indices.
  2. When constructing models and loading weights, also allow last PP rank to have token embeddings.
  3. Add dsa_topk_indices to the PP ring in scheduler_pp_mixin.

Accuracy Tests

GLM-5.2-NVFP4 on GB300 with Dynamo router.
Prefill PP4 + EAGLE(steps 1) with decode TP4 + EAGLE(steps 3).

GSM8K --max-tokens 16384:

prefill score
PP4 (this PR) 95.45%
TP4, no PP (control) 95.75%

Only the prefill's parallelism differs, and scores matches. AL on decode is 3.589, which looks healthy.

Unit tests (CPU, no GPU): 14 new cases + 9 subtests across
test/registered/unit/models/test_deepseek_pp_mtp_embedding.py and
TestPipelineParallelCompat in test/registered/unit/server_args/test_server_args.py.
Regression sweep over test/registered/unit/{spec,server_args,model_executor,models,disaggregation}:
1451 passed.

Speed Tests and Profiling

Not a performance change. Everything modified were previously unrunnable.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #34576894508
Latest PR Test (Extra): ❌ Run #34576894291
Latest PR Test (AMD ROCm 10): ❌ Run #34576894396

@@ -779,16 +778,20 @@ def _pp_prepare_tensor_dict(
tensor_dict["draft_topk_p"] = draft_input.topk_p.contiguous()
tensor_dict["draft_topk_index"] = draft_input.topk_index.contiguous()
tensor_dict["draft_hidden_states"] = draft_input.hidden_states.contiguous()
# The DSA IndexShare seed has to ride along for the same reason. It is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify this comment.

)
peer_info.kv_xfer_segments = prepared_segments

def _pp_layer_offset_dst_indices(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a docstring about this function: what does it do? what does returning None means? What does returning List[int] means?

n_src,
n_dst,
allow_positional_fallback=self.pp_size == 1,
dst_indices = self._pp_layer_offset_dst_indices(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this pattern:

out = doA()
if out is None:
    out = doB()

especially the names _pp_layer_offset_dst_indices() and build_transfer_entry_pairs() are not clear about why they are mutually exclusive.

Let's make it something like:

if isA:
    out = doA()
else:
    out = doB()

or just

out = doAorB()

where doAorB() automatically dispatches to the correct code path.

@@ -403,8 +403,13 @@ def do_load_weights(
# Skip loading extra bias for GPTQ models.
if name.endswith(".bias") and name not in params_dict:
continue
# Skip loading embed_tokens if not first rank in pipeline parallelism
if ".embed_tokens." in name and not self.pp_group.is_first_rank:
# Skip embed_tokens on a pipeline stage that has no embedding

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simplify this comment

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Codex] Review findings are included inline.

"Pipeline parallelism currently only supports EAGLE "
"(non-multi-layer) speculative decoding"
)
if get_platform().is_npu:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Codex] Severity: functional | Confidence: High

Issue

This leaves PP + EAGLE enabled for disaggregation_mode=decode and null off NPU, but non-last PP stages only skip draft execution in EAGLEWorkerV2's extend branch. A decode batch enters the other branch and reads self.draft_worker.draft_runner even though _hosts_draft left self._draft_worker as None, so the first decode fails instead of forwarding a proxy.

Fix

Restrict PP + speculative decoding to prefill mode on every platform until the non-last decode path is implemented and tested.

)
if cfg.speculative_algorithm is not None:
assert (
cfg.speculative_algorithm.upper() == "EAGLE"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Codex] Severity: functional | Confidence: High

Issue

This enables PP + EAGLE for every target model, but only DeepseekV2Model now materializes embed_tokens on the last stage. EagleDraftWorker.init_lm_head always calls the target's get_embed_and_head; for example, Llama keeps a PPMissingLayer on the last stage and LlamaForCausalLM.get_embed_and_head reads its missing .weight, so startup fails before serving.

Fix

Gate this support to the DeepSeek/NextN path, or generalize last-stage embedding ownership to every supported EAGLE target and add a non-DeepSeek regression test.

bonus_tokens=next_token_ids,
num_tokens_per_req=1,
num_tokens_for_logprob_per_req=1,
dsa_topk_indices=pp_outputs.tensors.get("draft_dsa_topk_indices"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Codex] Severity: style | Confidence: High

Issue

This serialization/rebuild is the fix for the reported DSA eager-versus-graph hang, but none of the new tests exercises it. The added tests cover argument validation, embedding ownership, and NIXL mapping; a missing or mismatched PP tensor key here would still pass all of them.

Fix

Extend test_generation_auxiliary_output.py with a draft input containing dsa_topk_indices. Assert that _pp_prepare_tensor_dict emits the seed and _pp_prep_batch_result restores the same tensor, plus the None case.

"""
placeholder = PPMissingLayer()
self.assertEqual(dict(placeholder.named_parameters()), {})
self.assertFalse(hasattr(placeholder, "weight"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Codex] Severity: style | Confidence: High

Issue

This added line uses hasattr, which is not allowed on modified lines. It also duplicates the stronger named_parameters() assertion immediately above it.

Fix

Remove this assertion and keep the explicit parameter-map check.



def pp_stage_needs_embedding(pp_group, speculative_algorithm) -> bool:
"""Does this pipeline stage need a real ``embed_tokens``?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Codex] Severity: style | Confidence: Medium

Issue

This small predicate has a long docstring that traces worker placement, checkpoint contents, weight-loader behavior, and head ownership. The new test module repeats the same explanation, making the code harder to scan and giving the rationale two places to drift.

Fix

Keep only the contract here, such as “The first stage embeds inputs; the last stage supplies the EAGLE draft embedding.” Keep the detailed investigation in the PR description.

@nvpohanh nvpohanh mentioned this pull request Sep 14, 2026
4 of 5 tasks
@YAMY1234 YAMY1234 self-assigned this Sep 14, 2026
@nvpohanh

Copy link
Copy Markdown
Collaborator

Let's track the updates in #39378 instead

@nvpohanh nvpohanh closed this Sep 14, 2026
@nvjullin
nvjullin deleted the pp-hetero-mtp branch November 4, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants