Skip to content
Closed
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
5ad325c
feat(sglang): add checkpoint-engine refit
tianyi-zhang-02 Aug 6, 2026
e8387b0
docs, CI wiring and a value-asserting alignment test
tianyi-zhang-02 Aug 7, 2026
fbc12ea
test(sglang): cover the refit gaps unit tests could not reach
tianyi-zhang-02 Aug 7, 2026
b1fe187
docs: reconcile refit.md with SGLang checkpoint-engine support
tianyi-zhang-02 Aug 7, 2026
ec69abe
fix(sglang): fail safely during checkpoint refit
tianyi-zhang-02 Aug 8, 2026
dc8defe
docs: record the dp_size restriction, and drop the now-dead metric field
tianyi-zhang-02 Aug 14, 2026
38c8d86
fix(sglang): reject pp_size>1 for checkpoint-engine refit
tianyi-zhang-02 Aug 14, 2026
29e6c3d
test(sglang): cover the pp_size guard
tianyi-zhang-02 Aug 14, 2026
fbf667f
fix(sglang): run checkpoint-engine refit inside a weight-update session
tianyi-zhang-02 Aug 28, 2026
2034cb9
fix(sglang): keep the refit receive loop off the Ray actor class
tianyi-zhang-02 Aug 28, 2026
c3a1687
refactor(sglang): type refit_cfg, correct the transport docs, drop de…
tianyi-zhang-02 Aug 28, 2026
6b83532
test(sglang): add a tp_size=2 checkpoint-engine functional test
tianyi-zhang-02 Aug 28, 2026
c963c83
fix(sglang): harden the refit envelope and its regression guards
tianyi-zhang-02 Aug 28, 2026
de6dbdd
test(sglang): give TP2 refit two policy senders
tianyi-zhang-02 Aug 30, 2026
9b47cc3
Merge remote-tracking branch 'upstream/main' into w3519
tianyi-zhang-02 Aug 30, 2026
5925bcf
fix(sglang): skip unused refit metadata gather
tianyi-zhang-02 Aug 30, 2026
dccfb5b
Merge current main into fix/sglang-checkpoint-engine-refit
tianyi-zhang-02 Sep 21, 2026
bfc5fc6
Merge current main (3519 validation refresh)
tianyi-zhang-02 Sep 21, 2026
9725291
test(sglang): align factory routing with Megatron support
tianyi-zhang-02 Sep 21, 2026
fa38884
Merge current main into SGLang checkpoint-engine refit
tianyi-zhang-02 Sep 30, 2026
cea34db
Merge current main into SGLang checkpoint-engine refit
tianyi-zhang-02 Sep 30, 2026
0034a72
chore(sglang): refresh checkpoint-engine refit on main
Kh4L Oct 19, 2026
d128f78
chore(sglang): refresh checkpoint-engine refit on current main
Kh4L Oct 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions docs/design-docs/checkpoint-engines.md
Original file line number Diff line number Diff line change
Expand Up @@ -358,15 +358,21 @@ To add a backend:
7. Add backend-specific config under `refit_cfg.<backend>`.
8. Use a `module:ClassName` `refit_transport` value, or add a short-name
mapping in `create_checkpoint_engine()` if the backend should be built in.
9. Run a non-colocated GRPO job and verify the `[vLLM refit]` timing line.
9. Run a non-colocated GRPO job and verify the per-refit timing line for the
generation backend under test (`[vLLM refit]` or `[SGLang refit]`).

Current limitations:

- Checkpoint-engine refit targets non-colocated policy-to-vLLM refit.
- SGLang and Megatron generation do not implement checkpoint-engine refit;
[issue #3288](https://github.com/NVIDIA-NeMo/RL/issues/3288) tracks
generation-side support. Megatron and DTensor policy backends are supported
when the generation backend is vLLM.
- Checkpoint-engine refit targets non-colocated policy-to-generation refit, for
the vLLM and SGLang generation backends.
- Megatron generation does not implement checkpoint-engine refit;
[issue #3288](https://github.com/NVIDIA-NeMo/RL/issues/3288) tracks the
remaining generation-side support. Megatron and DTensor policy backends are
supported for both vLLM and SGLang generation.
- SGLang refit requires each logical engine to fit on a single node
(`num_gpus_per_engine <= gpus_per_node`), requires `sglang_cfg.dp_size=1` and `sglang_cfg.pp_size=1`,
and does not support `shard_expert_weights=true`. All three are rejected
with an explicit error.
- The built-in NIXL backend uses paired policy-to-rollout transfer only.
- Sharded vLLM EP refit supports static expert ownership and canonical
unquantized Triton expert storage. Dynamic EPLB, redundant experts, and
Expand Down
17 changes: 14 additions & 3 deletions docs/guides/checkpoint-engine-refit.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,25 @@

Checkpoint-engine refit updates non-colocated generation workers directly from
policy workers. The built-in backend is NIXL, which can use UCX/RDMA for large
policy-to-vLLM refits.
policy-to-generation refits.

Use it only for non-colocated vLLM generation:
Use it only for non-colocated generation:

- `policy.generation.backend=vllm`
- `policy.generation.backend=vllm` or `policy.generation.backend=sglang`
- `policy.generation.colocated.enabled=false`
- `policy.generation.refit_transport=nixl`

| Generation backend | Supported | Notes |
| --- | --- | --- |
| vLLM | yes | Includes sharded EP refit (`shard_expert_weights`). |
| SGLang | yes | One node per logical engine, `dp_size=1`, `pp_size=1`; `shard_expert_weights` not supported. |
| Megatron | no | Tracked by [issue #3288](https://github.com/NVIDIA-NeMo/RL/issues/3288). |

For SGLang, NIXL terminates in the generation worker actor and the weights are
handed to the SGLang server subprocess over same-node CUDA IPC, reusing the same
`flattened_bucket` / `update_weights_from_tensor` contract as colocated refit.
The per-refit timing line is logged as `[SGLang refit]`.

Without checkpoint-engine refit, colocated vLLM uses ZMQ/CUDA IPC and
non-colocated vLLM uses NCCL. SGLang uses Ray CUDA IPC when colocated and its
own NCCL weight-update group with a Megatron policy when non-colocated.
Expand Down
15 changes: 8 additions & 7 deletions docs/guides/refit.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,16 @@ delta and NIXL cannot both be active.
| SGLang NCCL weight-update group | SGLang | Megatron | Trainer rank 0 broadcasts finalized HF tensors to the engine leaders. |
| NCCL reshard | vLLM | Megatron | Requires matching BF16 or blockwise FP8 precision; Megatron ETP must be 1. Currently supporting Megatron+vLLM backends. |
| Sparse delta | vLLM | Megatron | BF16/FP16, unquantized rollout only. |
| NIXL, full weights | vLLM | DTensor or Megatron | Supports the standard full-weight FP8 loader. DTensor FP8 KV-cache scale transfer is not yet supported. |
| NIXL, full weights | vLLM or SGLang | DTensor or Megatron | Supports the standard full-weight FP8 loader. DTensor FP8 KV-cache scale transfer is not yet supported. SGLang requires one node per logical engine, `dp_size=1` and `pp_size=1`. |
| NIXL, sharded experts | vLLM | DTensor or Megatron | Unquantized BF16/FP16 Triton MoE only; FP8/MXFP8 and dynamic expert placement are rejected. |

Non-colocated SGLang generation is supported with a Megatron policy and
`refit_transport: null`; it creates its own NCCL weight-update group. The NIXL
restrictions are on the generation backend; both Megatron and DTensor policy
workers can send weights. Sparse delta is currently limited to GRPO. NIXL is
initialized by the GRPO and distillation setup paths; PPO currently requires
colocated generation.
Non-colocated SGLang generation has two paths: a Megatron policy with
`refit_transport: null`, which creates its own NCCL weight-update group, and
checkpoint-engine refit (`refit_transport: nixl`), which works with either
policy backend. The NIXL restrictions are on the generation backend; both
Megatron and DTensor policy workers can send weights. Sparse delta is currently
limited to GRPO. NIXL is initialized by the GRPO and distillation setup paths;
PPO currently requires colocated generation.

## Minimal Configuration

Expand Down
23 changes: 18 additions & 5 deletions nemo_rl/algorithms/grpo.py
Original file line number Diff line number Diff line change
Expand Up @@ -1548,6 +1548,15 @@ def init_vllm_then_policy():

elif backend == "sglang":
generation_config = cast(SGLangConfig, generation_config)
refit_transport = generation_config.get("refit_transport")
if refit_transport is not None:
checkpoint_engine_config = checkpoint_engine_refit_config(generation_config)
if checkpoint_engine_config is None:
raise NotImplementedError(
f"SGLang does not support refit_transport={refit_transport!r}. "
"Use 'nixl', a custom checkpoint-engine path, or null for "
"colocated CUDA-IPC refit."
)

# Set model_path if not already set
if "model_path" not in generation_config["sglang_cfg"]:
Expand Down Expand Up @@ -1639,10 +1648,14 @@ def init_dynamo():

check_nccl_reshard_refit_support(master_config)

if generation_config.get("refit_transport") is not None and backend != "vllm":
if (
generation_config.get("refit_transport") is not None
and checkpoint_engine_config is None
and backend != "vllm"
):
raise NotImplementedError(
"Non-default refit transports are only supported for the vLLM "
f"generation backend, but policy.generation.backend={backend!r}. "
"This non-default refit transport is not supported for "
f"policy.generation.backend={backend!r}. "
"Set policy.generation.refit_transport=null. Support for other "
"generation backends is tracked in "
"https://github.com/NVIDIA-NeMo/RL/issues/3288."
Expand Down Expand Up @@ -1724,7 +1737,7 @@ def init_dynamo():
)
elif checkpoint_engine_config is not None:
t0 = time.perf_counter()
assert isinstance(policy_generation, VllmGeneration)
assert isinstance(policy_generation, (VllmGeneration, SGLangGeneration))
policy_generation.weight_synchronizer = create_weight_synchronizer(
policy=policy,
generation=policy_generation,
Expand All @@ -1734,7 +1747,7 @@ def init_dynamo():
inference_cluster=inference_cluster,
)
policy_generation.weight_synchronizer.init_communicator()
setup_timing_metrics.vllm_checkpoint_engine_init_time_s = (
setup_timing_metrics.extras[f"{backend}_checkpoint_engine_init_time_s"] = (
time.perf_counter() - t0
)
print(
Expand Down
1 change: 0 additions & 1 deletion nemo_rl/algorithms/metric_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ class SetupTimingMetrics:
teacher_reservation_time_s: Optional[float] = None
teacher_model_init_time_s: Optional[float] = None
teacher_init_time_s: Optional[float] = None
vllm_checkpoint_engine_init_time_s: Optional[float] = None

total_setup_time_s: Optional[float] = None
worker_setup_time_s: Optional[float] = None
Expand Down
Loading