Skip to content

feat: fix the vLLM DP path - #2517

Merged
terrykong merged 39 commits into
NVIDIA-NeMo:mainfrom
guyueh1:support_vllm_dp
Oct 29, 2026
Merged

terrykong merged 39 commits into
NVIDIA-NeMo:mainfrom
guyueh1:support_vllm_dp

Conversation

@guyueh1

@guyueh1 guyueh1 commented May 18, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Previously nemo-rl doesn't work for vllm's native DP (EP>TP), this PR wants to support this case.

The following basic tests have passed, now trying the nightly test

# eval test
uv run examples/run_eval.py \
generation.model_name=Qwen/Qwen3-30B-A3B \
cluster.num_nodes=2 \
cluster.gpus_per_node=4 \
generation.vllm_cfg.tensor_parallel_size=4 \
generation.vllm_cfg.expert_parallel_size=8 \
generation.vllm_cfg.async_engine=true \

# grpo test
uv run examples/run_grpo.py \
--config examples/configs/grpo_math_1B_megatron.yaml \
policy.model_name=Qwen/Qwen3-30B-A3B \
cluster.num_nodes=4 \
cluster.gpus_per_node=4 \
policy.generation.colocated.enabled=false \
policy.generation.colocated.resources.num_nodes=2 \
policy.generation.colocated.resources.gpus_per_node=4 \
policy.generation.vllm_cfg.tensor_parallel_size=4 \
policy.generation.vllm_cfg.expert_parallel_size=8 \
policy.generation.vllm_cfg.async_engine=true \
policy.megatron_cfg.expert_model_parallel_size=8 \
policy.sequence_packing.enabled=false \

New nightly test figures:
H100 with EP=8 async engine
image
https://wandb.ai/nvidia/nemo-rl/runs/4mcplb63

H100 with TP=2 EP=16 sync engine
image
https://wandb.ai/nvidia/nemo-rl/runs/b3mon8zg

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

Closes #1101

FP8 MoE refit scope

FP8 MoE refit scope: FlashInfer TRT-LLM FP8 MoE refit is not supported by this PR. Non-DeepSeek-V4 layouts that change weight shape or dtype fail explicitly rather than proceeding with an unvalidated refit. The Moonlight recipe uses moe_backend: triton; no FlashInfer TRT-LLM FP8 MoE refit GPU validation is claimed.

Signed-off-by: Guyue Huang <guyueh@login-lyris01.lyris.clusters.nvidia.com>
@guyueh1
guyueh1 requested review from a team as code owners May 18, 2026 04:46
@copy-pr-bot

copy-pr-bot Bot commented May 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@guyueh1
guyueh1 requested review from yuki-97 and removed request for a team May 18, 2026 04:46
@guyueh1 guyueh1 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label May 18, 2026
@guyueh1

guyueh1 commented May 18, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test efc6fc2

Guyue Huang added 2 commits May 18, 2026 09:36
Signed-off-by: Guyue Huang <guyueh@login-lyris01.lyris.clusters.nvidia.com>
Signed-off-by: Guyue Huang <guyueh@login-lyris01.lyris.clusters.nvidia.com>
@guyueh1

guyueh1 commented May 18, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9f381f2

@guyueh1

guyueh1 commented May 19, 2026

Copy link
Copy Markdown
Contributor Author

fast CI is failing for uuidgen: command not found, trying the full set to see if it helps

@guyueh1 guyueh1 added CI:L2 Run doctests, unit tests, functional tests, and convergence tests and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels May 19, 2026
@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 312e0ce

Signed-off-by: Alan Liu <alanliu@nvidia.com>
@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test f38d77a

@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

/ok to test f38d77a

@alanliu-nvidia, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test f38d77a

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

Thanks for sharing the nccl_reshard result and the fork PR. This PR currently validates native vLLM DP with the NCCL and checkpoint-engine refit transports; nccl_reshard has a separate EP==TP/EP==1 support guard and expert-mesh assumptions, so I do not want to remove that guard without an end-to-end train-to-vLLM refit/GRPO test on the new topology. I suggest tracking EP>TP nccl_reshard as a focused follow-up PR based on your work. The existing guard remains in place here; I am not claiming this PR supports that transport/topology.

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

GB200 FP8 refit follow-up (sync and async): both 4n4g jobs completed 3/3 GRPO steps with repeated weight refits and Slurm exit 0:0. Sync: job 8325533, W&B. Async: job 8325534, W&B. These are functional smoke tests, not long-horizon convergence evidence. The current PR head f38d77a63 has passed lint and up-to-date checks; full CI is still in the automated test approval queue.

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

Current-head GB200 validation update (SHA f38d77a63): all requested local GPU runs now completed successfully with NeMo-RL built-in W&B logging. FP8 TP1/EP4 sync and async each finished 3 GRPO steps with repeated refit (sync, async); these are functional smokes, not convergence claims. Non-colocated Qwen TP4/EP8 completed 10 steps with median token-mult error 1.00752 < 1.02 (W&B); single-controller R3/forward/hash completed 3 steps, median 1.00962 < 1.02, 0 hash mismatches, and strict check_r3_trace.py --require-forward-verify --require-cp-identity passed on 25,252 records (W&B). Two-instance checkpoint-engine NIXL CPU staging completed 3 steps with repeated 5,344-tensor refits (W&B); this does not validate CUDA VRAM staging. Sync TP1/EP16 with 8 requests across 16 ranks completed a GRPO step without empty-shard deadlock (W&B). Moonlight DP8 and TP2/EP16 each completed 10 steps, saved the final checkpoint, and passed all four nightly metric checks, including steps-5–9 timing means 38.54s < 70s (DP8 W&B) and 69.58s < 100s (TP2/EP16 W&B); TP2/EP16 had one 157.95s slow step included in that mean. These GB200 8n4g/HybridEP scaled runs do not replace the exact H100/DeepEP nightly. Current-head full CI is now in container builds. DCO sign-offs for two commits are pending the already-authorized post-CI history rewrite, after which the new SHA will need CI again.

Signed-off-by: Alan Liu <alanliu@nvidia.com>
@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test c465781

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

CI follow-up: the prior L2 run (f38d77a63) built both H100/GB200 containers and passed docs, lint, and 26/27 unit groups, but L0_Unit_Tests_Other found one PR-introduced test-double regression: the reshard fake omitted VllmGeneration.vllm_dp_size (5,792 other tests passed in that group). This is fixed in the current head c46578113134e136c33a70d3c22da82c3b91bc0c by explicitly setting the fake to 1 and testing that internal DP>1 rejects partial-shard rebuild. The two older unsigned commits now have DCO sign-offs; GitHub DCO is green. I posted /ok to test for the new SHA. Full functional/nightly CI on this SHA is still pending, so this is not merge-ready yet.

Signed-off-by: Alan Liu <alanliu@nvidia.com>
@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test a6c8c3a

@alanliu-nvidia

alanliu-nvidia commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

@andresnowak Thanks for the nccl_reshard pointer. I opened focused draft follow-up #4941, stacked on this PR rather than expanding #2517. On GB200, TP1/EP4 with two native-DP4 vLLM instances completed 3 GRPO steps and repeated nccl_reshard refits in both async (https://wandb.ai/nvidia/dpvllm/runs/9hcthmz9; token-mult error 1.00745/1.00732/1.00966) and sync (https://wandb.ai/nvidia/dpvllm/runs/e3sn8dt7; 1.00733/1.00843/1.00849). Both Slurm jobs completed 0:0. This container uses the xferdtensor_python exact-transfer fallback because nccl.m2n.reshard is absent, so I am not claiming the native m2n kernel was tested. #4941 remains draft, pending copy-pr-bot vetting/CI and rebase after this PR merges.

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

CI update for current head a6c8c3aaa42efb8ba78b42aebb93e020496960f1: L2 run 40427554640 completed successfully. All L0 groups passed. The H100 L1_Functional_Tests_Megatron_Generation job failed once after finishing its two training steps, then passed on the same SHA when rerun (attempt 3); the GB200 counterpart also passed. I did not change the unrelated Megatron code. The CI quality check is successful on attempt 4. GitHub still shows Coverage (e2e) in progress even though all of its job steps report success, so I am monitoring that final check before calling the PR fully green. The branch is currently mergeable; the draft nccl_reshard follow-up remains separate in #4941.

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 4e10de3

terrykong
terrykong previously approved these changes Oct 27, 2026
Signed-off-by: Alan Liu <alanliu@nvidia.com>
@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 8610bbe

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

CI unit-test fix follow-up: the two native-DP empty-shard test fakes now initialize _native_lora_request = None, matching the worker constructor. Both previously failing tests passed on each of four GB200 nodes (Slurm job 8414499, 8/8 targeted test executions; W&B run and log artifact). Commit 8610bbe9c is DCO-signed; requested full CI for this SHA. The local four-node result is targeted unit-test evidence, not a claim that full CI has passed.

@alanliu-nvidia

Copy link
Copy Markdown
Contributor

/ok to test dbed9e4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L0 Run doctests and unit tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support use vLLM DP + EP in async engine

5 participants