Skip to content

feat(sglang): Megatron+SGLang + MXFP8 - #3091

Closed
xiuhu17 wants to merge 10 commits into
NVIDIA-NeMo:mainfrom
xiuhu17:zhw/mxfp8_support_main
Closed

xiuhu17 wants to merge 10 commits into
NVIDIA-NeMo:mainfrom
xiuhu17:zhw/mxfp8_support_main

Conversation

@xiuhu17

@xiuhu17 xiuhu17 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Ports the megatron + sglang weight-update scheme into the main-tracking branch: Megatron trainers export HF-named tensors through AutoBridge, optionally quantize them to MXFP8 online, and push them to colocated SGLang engines via Ray CUDA-IPC (weight_transfer_mode: ipc) or to disaggregate engines via a trainer-rank-0 NCCL broadcast group (weight_transfer_mode: broadcast). The FSDP colocated path is unified onto the same connect/send machinery, replacing the previous HTTP streaming design.

mxfp8 checkpoint conversion, weight update and fine-grained layer precision is adopted from:
radixark/miles#615 which is done by @zianglih.

What changed

New scheme components

  • mxfp8_quantization_core.py — single source of truth for per-tensor MXFP8 decisions (should_quantize / quantize_mxfp8, flashinfer swizzle-free kernel, UE8M0 scales, [1,32] blocks), shared by offline conversion and online refit
  • mxfp8_setup.py — offline HF→MXFP8 checkpoint conversion with hash-qualified caching; ensure_mxfp8_checkpoint bootstraps SGLang from a quantized checkpoint at setup
  • megatron_sglang_weight_iterator.py — walks bridge.export_hf_weights, applies the MXFP8 policy per tensor (expanding (weight, weight_scale_inv) pairs), and packs buckets sized by post-quantization bytes
  • fault_tolerance.py — RolloutHealthMonitor daemon that health-checks engines and restarts dead actors; recovered engines are reconnected on the next refit via get_updatable_engines_and_lock / recover_updatable_engines

Refit plumbing (policy/utils.py)

  • connect_colocate_topology — layout-keyed per-engine Gloo gather subgroups, supporting explicit engine_gpu_counts / engine_gpu_offsets and rebuild-on-recover
  • send_hf_buckets_via_ipc_actor_impl — gather-to-src + Ray actor RPC with per-chunk ray.get (IPC lifetime safety) and engine-result validation
  • broadcast_hf_buckets_via_distributed_impl + NCCL group connect/disconnect helpers — trainer-rank-0 broadcast, serialized against engine recovery by a Ray Lock actor

Drivers and dispatch

  • refit_sglang_{colocated,distributed} drivers on both backends with the full lifecycle: recover → connect → pause_generation(mode) → invalidate_kv_cache → send → post_process_weights → continue_generation
  • grpo._refit_sglang_dispatch routes by trainer backend and transfer mode; weight_transfer_mode is validated against colocated.enabled; also fixes buffer_size_bytes being undefined on the non-colocated SGLang path
  • SGLang worker gains init/destroy_weights_update_group, update_weights_from_distributed, pause/continue_generation, post_process_weights, get_weight_version

FSDP path unified (HTTP streaming removed)

  • stream_weights_via_http_impl, _ensure_ipc_topology, _flush_bucket, set_rollout_num_gpus_per_engine are removed; the DTensor worker now uses the same connect_sglang_rollout_engines + send_hf_buckets_via_ipc_actor_impl path (BF16-only)
  • HTTPWeightSynchronizer → SGLangColocatedWeightSynchronizer, delegating to the backend refit drivers

Config

  • SglangQuantizationConfig (quantization.scheme: bf16|mxfp8 + conversion knobs), fault-tolerance keys, and sglang_server_config.weight_transfer_mode, with defaults documented in grpo_math_1B_sglang.yaml
  • NCCL_CUMEM_ENABLE=0 aligned between trainer setup (megatron/automodel) and SGLang engine actors

Testing

  • uv run ruff check / ruff format --check clean across the diff
  • tests/unit/weight_sync/ rewritten against the dispatch seam (mock refit_sglang_colocated)
  • GPU end-to-end (colocated ipc + broadcast, bf16 + mxfp8) verified on the internal SGLang e2e container; megatron+sglang unit tests to follow in a separate test PR

@xiuhu17
xiuhu17 requested review from a team as code owners July 7, 2026 01:21
@copy-pr-bot

copy-pr-bot Bot commented Jul 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

xiuhu17 added 9 commits July 8, 2026 19:13
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
@xiuhu17
xiuhu17 force-pushed the zhw/mxfp8_support_main branch from 94bd22c to acb2fe7 Compare July 9, 2026 02:15
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Jul 9, 2026
@Kh4L

Kh4L commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

I implemented and ran this PR's disaggregated weight_transfer_mode: broadcast design on a GB200 NVL72 cluster (Megatron trainer node + separate SGLang engine node, stock sglang==0.5.12.post1, torch 2.11.0 / NCCL 2.28.9 on both sides) as part of SGLang enablement work on main (#3127 and NVIDIA-NeMo/Gym#1787), with a connect_rollout_engines_from_distributed equivalent that is functionally identical to this branch's

Findings that likely affect this PR on GB200:

The trainer-rank-0 weight-update group never completes NCCL init on GB200 NVL72. Rendezvous and store are healthy, all ranks join, both sides reach Connected all rings - then init collapses at the bootstrap finalize (bootstrap.cc allgather: root closes its connections, members see socketProgress: Connection closed by remote peer). Reproduces identically across every topology I tried: cross-rack (where the cluster's forced NCCL_MNNVL_ENABLE=1 makes NCCL build impossible cross-clique MNNVL channels), cross-rack with MNNVL disabled, and same-rack/same-clique where MNNVL is genuinely valid.

Same NCCL 2.28.9+cuda13.0 on both ends. I have full NCCL_DEBUG=INFO (SUBSYS=INIT,BOOTSTRAP) traces from all ranks if useful.
Question: have you broadcast mode been run on ARM/GB200/NVL72 yet, or only on x86 so far?

TCPStore master lifetime (from torch): the NCCL backend is constructed lazily, so after the connect call returns, nothing references the TCPStore master created inside init_process_group. I observed the store listener getting GC'd with the stack frame, after which every member fails store->get("0") with Failed to recv, got 0 bytes at the first collective. Pinning the store (return it alongside the pg, keep a ref on the trainer worker) fixed it for me.
Worth doing here too unless something else on this branch keeps it alive.

A question: how are engine-side receives dispatched during a broadcast refit here?
In my port, receive tasks dispatched via engine-actor remote calls never got scheduled while a refit was in flight (driving /update_weights_from_distributed directly over HTTP worked); if this branch goes through the engine actors, worth checking under an active refit.

That's in the context of upstreaming the async-GRPO loop side of SGLang enablement in #3127 and the gym-side token-exact proxy in NVIDIA-NeMo/Gym#1787 (this PR's transfer scheme plus those two compose into the full SGLang training path).

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Jul 11, 2026
@Kh4L

Kh4L commented Jul 13, 2026 •

Copy link
Copy Markdown
Contributor

Update on the previous GB200 message: I root-caused it, and the mechanism differs from what I first described. There was no five-rank communicator collapse; no five-rank comm was ever created.

Agent assisted explanation:

Root cause: an ncclCommSplit auto-split, not a transport failure. On my training base the trainer default process group is initialized device-bound (device_id=). With a device-bound default group, PyTorch 2.11's _new_process_group_helper(..., global_ranks_in_group=[]) auto-sets Options.split_from to the existing trainer comm. Only trainer rank 0 calls the refit helper, so the first collective invokes ncclCommSplit on the trainer parent while ranks 1-3 never enter the split. Rank 0 blocks in the parent bootstrap all-gather (init.cc:1465 -> 6, commGetSplitInfo in NCCL 2.28.9); the rest watchdog. Topology-independent, peer-close originates trainer-side.

Relevance here: this predicate only fires when the default PG is device-bound. Current upstream main inits the Megatron default PG without device_id, and this PR's branch is the same, so connect_rollout_engines_from_distributed should not hit this as-is. Flagging it because (1) any base that binds the default group to a device (some trainer setups do) silently trips the copied init_process_group([...]) pattern, and (2) the store-lifetime point from my earlier note still applies regardless.

Fix: build an independent ProcessGroupNCCL without the split-selecting helper (fresh Options, no split_from, full c10d state registration, pin the prefixed store).

I can to open it as a standalone PR you can build on/add to this PR

@xiuhu17

xiuhu17 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

@Kh4L Thanks for finding this issue and huge thanks for finding the root cause! At this stage, I am going to split it into four prs and we could open the fifth for this.

Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
@xiuhu17

xiuhu17 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

@Kh4L Hi, I just fix this issue. I am going to

  1. rebase
  2. close this pr
  3. open four prs: mxfp8, megatron, fault tolerance, current hang fix(I could close this one and let you open this pr).

What do you think?

@xiuhu17 xiuhu17 closed this Jul 14, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants