Repository navigation
Conversation
| @@ -779,16 +778,20 @@ def _pp_prepare_tensor_dict( | |||
| tensor_dict["draft_topk_p"] = draft_input.topk_p.contiguous() | |||
| tensor_dict["draft_topk_index"] = draft_input.topk_index.contiguous() | |||
| tensor_dict["draft_hidden_states"] = draft_input.hidden_states.contiguous() | |||
| # The DSA IndexShare seed has to ride along for the same reason. It is | |||
| ) | ||
| peer_info.kv_xfer_segments = prepared_segments | ||
|
|
||
| def _pp_layer_offset_dst_indices( |
There was a problem hiding this comment.
add a docstring about this function: what does it do? what does returning None means? What does returning List[int] means?
| n_src, | ||
| n_dst, | ||
| allow_positional_fallback=self.pp_size == 1, | ||
| dst_indices = self._pp_layer_offset_dst_indices( |
There was a problem hiding this comment.
I don't like this pattern:
out = doA()
if out is None:
out = doB()
especially the names _pp_layer_offset_dst_indices() and build_transfer_entry_pairs() are not clear about why they are mutually exclusive.
Let's make it something like:
if isA:
out = doA()
else:
out = doB()
or just
out = doAorB()
where doAorB() automatically dispatches to the correct code path.
| @@ -403,8 +403,13 @@ def do_load_weights( | |||
| # Skip loading extra bias for GPTQ models. | |||
| if name.endswith(".bias") and name not in params_dict: | |||
| continue | |||
| # Skip loading embed_tokens if not first rank in pipeline parallelism | |||
| if ".embed_tokens." in name and not self.pp_group.is_first_rank: | |||
| # Skip embed_tokens on a pipeline stage that has no embedding | |||
nvpohanh
left a comment
There was a problem hiding this comment.
[by Codex] Review findings are included inline.
| "Pipeline parallelism currently only supports EAGLE " | ||
| "(non-multi-layer) speculative decoding" | ||
| ) | ||
| if get_platform().is_npu: |
There was a problem hiding this comment.
[by Codex] Severity: functional | Confidence: High
Issue
This leaves PP + EAGLE enabled for disaggregation_mode=decode and null off NPU, but non-last PP stages only skip draft execution in EAGLEWorkerV2's extend branch. A decode batch enters the other branch and reads self.draft_worker.draft_runner even though _hosts_draft left self._draft_worker as None, so the first decode fails instead of forwarding a proxy.
Fix
Restrict PP + speculative decoding to prefill mode on every platform until the non-last decode path is implemented and tested.
| ) | ||
| if cfg.speculative_algorithm is not None: | ||
| assert ( | ||
| cfg.speculative_algorithm.upper() == "EAGLE" |
There was a problem hiding this comment.
[by Codex] Severity: functional | Confidence: High
Issue
This enables PP + EAGLE for every target model, but only DeepseekV2Model now materializes embed_tokens on the last stage. EagleDraftWorker.init_lm_head always calls the target's get_embed_and_head; for example, Llama keeps a PPMissingLayer on the last stage and LlamaForCausalLM.get_embed_and_head reads its missing .weight, so startup fails before serving.
Fix
Gate this support to the DeepSeek/NextN path, or generalize last-stage embedding ownership to every supported EAGLE target and add a non-DeepSeek regression test.
| bonus_tokens=next_token_ids, | ||
| num_tokens_per_req=1, | ||
| num_tokens_for_logprob_per_req=1, | ||
| dsa_topk_indices=pp_outputs.tensors.get("draft_dsa_topk_indices"), |
There was a problem hiding this comment.
[by Codex] Severity: style | Confidence: High
Issue
This serialization/rebuild is the fix for the reported DSA eager-versus-graph hang, but none of the new tests exercises it. The added tests cover argument validation, embedding ownership, and NIXL mapping; a missing or mismatched PP tensor key here would still pass all of them.
Fix
Extend test_generation_auxiliary_output.py with a draft input containing dsa_topk_indices. Assert that _pp_prepare_tensor_dict emits the seed and _pp_prep_batch_result restores the same tensor, plus the None case.
| """ | ||
| placeholder = PPMissingLayer() | ||
| self.assertEqual(dict(placeholder.named_parameters()), {}) | ||
| self.assertFalse(hasattr(placeholder, "weight")) |
There was a problem hiding this comment.
[by Codex] Severity: style | Confidence: High
Issue
This added line uses hasattr, which is not allowed on modified lines. It also duplicates the stronger named_parameters() assertion immediately above it.
Fix
Remove this assertion and keep the explicit parameter-map check.
|
|
||
|
|
||
| def pp_stage_needs_embedding(pp_group, speculative_algorithm) -> bool: | ||
| """Does this pipeline stage need a real ``embed_tokens``? |
There was a problem hiding this comment.
[by Codex] Severity: style | Confidence: Medium
Issue
This small predicate has a long docstring that traces worker placement, checkpoint contents, weight-loader behavior, and head ownership. The new test module repeats the same explanation, making the code harder to scan and giving the rationale two places to drift.
Fix
Keep only the contract here, such as “The first stage embeds inputs; the last stage supplies the EAGLE draft embedding.” Keep the detailed investigation in the PR description.
|
Let's track the updates in #39378 instead |
Motivation
This PR is based on #38853, which fixes disagg PP for MLA. This PR fixes disagg PP + MTP for GLM5.2.
There are hard errors when running PP prefill + DP decode on GLM5.2, caused by three things:
check_server_argsinarg_groups/validation_hook.pyand by_assert_pp_mtp_compatinmodel_executor/model_runner_components/layer_setup.py. The runtime underneath is already PP-aware and platform-neutral:EAGLEWorkerV2builds the draft on the last PP stage only (_hosts_draft) and every earlier stage forwards the target's PP proxy tensors and returns.scheduler_pp_mixindrops the draft model's DSA IndexShare seed (dsa_topk_indices), so decode has to regenerate them. However, decode DP currently hangs if one rank receives nothing which will run cudagraph, and another rank receives a request with no topk indices which runs eagerly.On (3): the last PP stage's draft proposal crosses the pipeline ring as
draft_topk_p/draft_topk_index/draft_hidden_states(_pp_prepare_tensor_dict) and is rebuilt into a freshEagleDraftInputon every rank, which is then rebound tobatch.spec_info.Modifications
validation_hookkeeps the overlap-schedule and EAGLE-shape restrictions for all platforms and keeps NPU's extra prefill-only rule, but the platform branch is gone. Removed_assert_pp_mtp_compatandspec_algorithmargument toresolve_layer_indices.dsa_topk_indicesto the PP ring inscheduler_pp_mixin.Accuracy Tests
GLM-5.2-NVFP4 on GB300 with Dynamo router.
Prefill PP4 + EAGLE(steps 1) with decode TP4 + EAGLE(steps 3).
GSM8K
--max-tokens 16384:PP4(this PR)TP4, no PP (control)Only the prefill's parallelism differs, and scores matches. AL on decode is 3.589, which looks healthy.
Unit tests (CPU, no GPU): 14 new cases + 9 subtests across
test/registered/unit/models/test_deepseek_pp_mtp_embedding.pyandTestPipelineParallelCompatintest/registered/unit/server_args/test_server_args.py.Regression sweep over
test/registered/unit/{spec,server_args,model_executor,models,disaggregation}:1451 passed.
Speed Tests and Profiling
Not a performance change. Everything modified were previously unrunnable.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #34576894508
Latest PR Test (Extra): ❌ Run #34576894291
Latest PR Test (AMD ROCm 10): ❌ Run #34576894396