Repository navigation
Conversation
94bd22c to
acb2fe7
Compare
|
I implemented and ran this PR's disaggregated Findings that likely affect this PR on GB200: The trainer-rank-0 weight-update group never completes NCCL init on GB200 NVL72. Rendezvous and store are healthy, all ranks join, both sides reach Same TCPStore master lifetime (from torch): the NCCL backend is constructed lazily, so after the connect call returns, nothing references the TCPStore master created inside A question: how are engine-side receives dispatched during a broadcast refit here? That's in the context of upstreaming the async-GRPO loop side of SGLang enablement in #3127 and the gym-side token-exact proxy in NVIDIA-NeMo/Gym#1787 (this PR's transfer scheme plus those two compose into the full SGLang training path). |
|
Update on the previous GB200 message: I root-caused it, and the mechanism differs from what I first described. There was no five-rank communicator collapse; no five-rank comm was ever created. Agent assisted explanation:
I can to open it as a standalone PR you can build on/add to this PR |
|
@Kh4L Thanks for finding this issue and huge thanks for finding the root cause! At this stage, I am going to split it into four prs and we could open the fifth for this. |
|
@Kh4L Hi, I just fix this issue. I am going to
What do you think? |
Summary
Ports the megatron + sglang weight-update scheme into the main-tracking branch: Megatron trainers export HF-named tensors through AutoBridge, optionally quantize them to MXFP8 online, and push them to colocated SGLang engines via Ray CUDA-IPC (
weight_transfer_mode: ipc) or to disaggregate engines via a trainer-rank-0 NCCL broadcast group (weight_transfer_mode: broadcast). The FSDP colocated path is unified onto the same connect/send machinery, replacing the previous HTTP streaming design.mxfp8 checkpoint conversion, weight update and fine-grained layer precision is adopted from:
radixark/miles#615 which is done by @zianglih.
What changed
New scheme components
mxfp8_quantization_core.py— single source of truth for per-tensor MXFP8 decisions (should_quantize/quantize_mxfp8, flashinfer swizzle-free kernel, UE8M0 scales, [1,32] blocks), shared by offline conversion and online refitmxfp8_setup.py— offline HF→MXFP8 checkpoint conversion with hash-qualified caching;ensure_mxfp8_checkpointbootstraps SGLang from a quantized checkpoint at setupmegatron_sglang_weight_iterator.py— walksbridge.export_hf_weights, applies the MXFP8 policy per tensor (expanding(weight, weight_scale_inv)pairs), and packs buckets sized by post-quantization bytesfault_tolerance.py—RolloutHealthMonitordaemon that health-checks engines and restarts dead actors; recovered engines are reconnected on the next refit viaget_updatable_engines_and_lock/recover_updatable_enginesRefit plumbing (
policy/utils.py)connect_colocate_topology— layout-keyed per-engine Gloo gather subgroups, supporting explicitengine_gpu_counts/engine_gpu_offsetsand rebuild-on-recoversend_hf_buckets_via_ipc_actor_impl— gather-to-src + Ray actor RPC with per-chunkray.get(IPC lifetime safety) and engine-result validationbroadcast_hf_buckets_via_distributed_impl+ NCCL group connect/disconnect helpers — trainer-rank-0 broadcast, serialized against engine recovery by a RayLockactorDrivers and dispatch
refit_sglang_{colocated,distributed}drivers on both backends with the full lifecycle: recover → connect →pause_generation(mode)→invalidate_kv_cache→ send →post_process_weights→continue_generationgrpo._refit_sglang_dispatchroutes by trainer backend and transfer mode;weight_transfer_modeis validated againstcolocated.enabled; also fixesbuffer_size_bytesbeing undefined on the non-colocated SGLang pathinit/destroy_weights_update_group,update_weights_from_distributed,pause/continue_generation,post_process_weights,get_weight_versionFSDP path unified (HTTP streaming removed)
stream_weights_via_http_impl,_ensure_ipc_topology,_flush_bucket,set_rollout_num_gpus_per_engineare removed; the DTensor worker now uses the sameconnect_sglang_rollout_engines+send_hf_buckets_via_ipc_actor_implpath (BF16-only)HTTPWeightSynchronizer→SGLangColocatedWeightSynchronizer, delegating to the backend refit driversConfig
SglangQuantizationConfig(quantization.scheme: bf16|mxfp8+ conversion knobs), fault-tolerance keys, andsglang_server_config.weight_transfer_mode, with defaults documented ingrpo_math_1B_sglang.yamlNCCL_CUMEM_ENABLE=0aligned between trainer setup (megatron/automodel) and SGLang engine actorsTesting
uv run ruff check/ruff format --checkclean across the difftests/unit/weight_sync/rewritten against the dispatch seam (mockrefit_sglang_colocated)