Repository navigation
feat: pause SingleController refits and enable KV recomputation by default - #3929
Conversation
|
/ok to test e16a857 |
|
Real-GPU validation completed at exact PR head I ran the full-size native SingleController vLLM GRPO recipe (
The only traceback in each log was a post-success W&B service |
e16a857 to
daa045d
Compare
|
/ok to test daa045d |
|
Addressed the The failure was PR-caused but test-only: Validation:
|
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
daa045d to
15074e4
Compare
|
Addressed the remaining The failure was PR-caused but test-only: Validation:
|
|
/ok to test 15074e4 |
youngeunkwon0405
left a comment
There was a problem hiding this comment.
Team review: 4 agents (rl-expert, test-agent, bug-finder, devil's advocate) analyzed this PR; every finding below was independently re-verified against source, including the pinned vLLM 0.25.1 wheel.
Verified clean (no action): clear_cache-at-pause genuinely subsumes post-refit invalidation (vLLM keep-mode pause_scheduler resets caches at pause); _refit_leader_workers() targets exactly the set the old run_rank_0_only_axes dispatch selected (incl. pp>1) and matches the existing refit dispatches; interface default hooks keep unsupported backends safe; fail-stop resume semantics mirror trajectory_collector; removing the drain-gate TODO is consistent (docs + issue #2625 still track it). Ruff 0.9.9 check+format clean. The A/B W&B evidence (preserve vs recompute, 11 pause/refit/resume cycles each) is solid.
Findings: 1 hardening suggestion, 1 docs note, 2 optional test suggestions (both verified passing locally on CPU) — all inline.
Generated by Claude Code
…roller-refit-pause Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test 1f68285 |
|
Post-review validation for exact head 1f68285:
Claude review: all four threads were addressed in 1f68285, replied to, and resolved: bounded lifecycle calls, timeout-budget documentation, recovery ordering coverage, and pause-failure coverage. Exact-head CI was retriggered; branch-up-to-date, lint, docs, and script-coverage checks are passing, with container builds still running. |
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test 463f247 |
|
Simplified the post-review implementation at exact head
Validation at this exact head:
The only post-success traceback was the same W&B service |
|
/ok to test 463f247 |
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test df75b8c |
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test d001be0 |
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test 9c15e56 |
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test 064a49e |
|
/ok to test 064a49e |
|
/ok to test 064a49e |
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
|
/ok to test 61dfb29 |
|
H100 Megatron nightly results on The original
The median includes all 500 steps, including the 55.0465s first step. Each required metric has exactly 500 finite values for steps 1–500. The W&B run is finished and independently contains all 500 steps. Post-processing qualification: the original training allocation ended Router validation: the full-mode These two-GPU runs validate router pass-through, not failover/recovery: scenarios requiring at least three GPUs were explicitly skipped. Update: overall L1 CI, attempt 2 completed SUCCESS on the same PR head. The previously failing H100 Megatron-generation, GB200 SFT, and GB200 SingleController/Gym lanes all passed on retry without source changes. The CI-quality gate also passed. The original failures remain preserved; retry success does not establish the exact cause of each original failure. |
RL#4719 implemented the SingleController's refit pause for Dynamo as a drain, because Dynamo then reloaded weights layerwise through its own path. Since the refit moved into NeMo RL's vLLM worker extension, each refit is one update_weights_from_distributed call per engine, the same update the vLLM backend takes. The drain held every refit until each running request finished. - DynamoGeneration.pause_generation_for_refit now freezes every engine through Dynamo's pause_generation route with vLLM's keep mode, as the vLLM backend pauses its own engines (#3929): frozen requests continue on the new weights after the resume, and clear_cache (recompute_kv_cache_after_weight_updates, the default) drops their KV so they recompute it. timeout_s bounds each engine's pause and resume. - flush_cache keeps its drain for the legacy loop's post-refit invalidation. - Drop the control_timeout_s >= request_timeout_s check: nothing on the SingleController path waits for running requests any more. - The functional test counts a keep-mode pause and a resume per sync. Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on top of this PR, before the keep-mode pause (drained job 3069784 vs serve-through job 3074960): - post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s - max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error 0.00064 -> 0.00072 (gates 1.05 / 0.002) - steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is generation-bound, so the drain overlapped the trainer's wait for rollouts (exposed_generation ~0 -> ~71 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
vLLM's wait pause drains only running requests and never starts a waiting one, so a decode request still receiving its prompt KV keeps its blocks and the cache reset that completes the pause fails. Found on GB200 (tyche): the second refit of the P/D SingleController recipe failed in reset_prefix_cache. After every prefill engine has drained, poll each decode engine's vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps up with 512 rollouts per step. The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in place, but vLLM's cache reset raises while a decode engine still waits for remote KV or a prefill engine still holds blocks it handed off. So a disaggregated fleet drains in this order when the pause clears caches (recompute_kv_cache_after_weight_updates, the default), and freezes in the same order otherwise. control_timeout_s must then cover request_timeout_s, since it bounds the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
RL#4719 implemented the SingleController's refit pause for Dynamo as a drain, because Dynamo then reloaded weights layerwise through its own path. Since the refit moved into NeMo RL's vLLM worker extension, each refit is one update_weights_from_distributed call per engine, the same update the vLLM backend takes. The drain held every refit until each running request finished. - DynamoGeneration.pause_generation_for_refit now freezes every engine through Dynamo's pause_generation route with vLLM's keep mode, as the vLLM backend pauses its own engines (NVIDIA-NeMo#3929): frozen requests continue on the new weights after the resume, and clear_cache (recompute_kv_cache_after_weight_updates, the default) drops their KV so they recompute it. timeout_s bounds each engine's pause and resume. - flush_cache keeps its drain for the legacy loop's post-refit invalidation. - Drop the control_timeout_s >= request_timeout_s check: nothing on the SingleController path waits for running requests any more. - The functional test counts a keep-mode pause and a resume per sync. Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on top of this PR, before the keep-mode pause (drained job 3069784 vs serve-through job 3074960): - post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s - max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error 0.00064 -> 0.00072 (gates 1.05 / 0.002) - steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is generation-bound, so the drain overlapped the trainer's wait for rollouts (exposed_generation ~0 -> ~71 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
vLLM's wait pause drains only running requests and never starts a waiting one, so a decode request still receiving its prompt KV keeps its blocks and the cache reset that completes the pause fails. Found on GB200 (tyche): the second refit of the P/D SingleController recipe failed in reset_prefix_cache. After every prefill engine has drained, poll each decode engine's vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps up with 512 rollouts per step. The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in place, but vLLM's cache reset raises while a decode engine still waits for remote KV or a prefill engine still holds blocks it handed off. So a disaggregated fleet drains in this order when the pause clears caches (recompute_kv_cache_after_weight_updates, the default), and freezes in the same order otherwise. control_timeout_s must then cover request_timeout_s, since it bounds the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
RL#4719 implemented the SingleController's refit pause for Dynamo as a drain, because Dynamo then reloaded weights layerwise through its own path. Since the refit moved into NeMo RL's vLLM worker extension, each refit is one update_weights_from_distributed call per engine, the same update the vLLM backend takes. The drain held every refit until each running request finished. - DynamoGeneration.pause_generation_for_refit now freezes every engine through Dynamo's pause_generation route with vLLM's keep mode, as the vLLM backend pauses its own engines (NVIDIA-NeMo#3929): frozen requests continue on the new weights after the resume, and clear_cache (recompute_kv_cache_after_weight_updates, the default) drops their KV so they recompute it. timeout_s bounds each engine's pause and resume. - flush_cache keeps its drain for the legacy loop's post-refit invalidation. - Drop the control_timeout_s >= request_timeout_s check: nothing on the SingleController path waits for running requests any more. - The functional test counts a keep-mode pause and a resume per sync. Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on top of this PR, before the keep-mode pause (drained job 3069784 vs serve-through job 3074960): - post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s - max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error 0.00064 -> 0.00072 (gates 1.05 / 0.002) - steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is generation-bound, so the drain overlapped the trainer's wait for rollouts (exposed_generation ~0 -> ~71 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
vLLM's wait pause drains only running requests and never starts a waiting one, so a decode request still receiving its prompt KV keeps its blocks and the cache reset that completes the pause fails. Found on GB200 (tyche): the second refit of the P/D SingleController recipe failed in reset_prefix_cache. After every prefill engine has drained, poll each decode engine's vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps up with 512 rollouts per step. The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in place, but vLLM's cache reset raises while a decode engine still waits for remote KV or a prefill engine still holds blocks it handed off. So a disaggregated fleet drains in this order when the pause clears caches (recompute_kv_cache_after_weight_updates, the default), and freezes in the same order otherwise. control_timeout_s must then cover request_timeout_s, since it bounds the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
RL#4719 implemented the SingleController's refit pause for Dynamo as a drain, because Dynamo then reloaded weights layerwise through its own path. Since the refit moved into NeMo RL's vLLM worker extension, each refit is one update_weights_from_distributed call per engine, the same update the vLLM backend takes. The drain held every refit until each running request finished. - DynamoGeneration.pause_generation_for_refit now freezes every engine through Dynamo's pause_generation route with vLLM's keep mode, as the vLLM backend pauses its own engines (NVIDIA-NeMo#3929): frozen requests continue on the new weights after the resume, and clear_cache (recompute_kv_cache_after_weight_updates, the default) drops their KV so they recompute it. timeout_s bounds each engine's pause and resume. - flush_cache keeps its drain for the legacy loop's post-refit invalidation. - Drop the control_timeout_s >= request_timeout_s check: nothing on the SingleController path waits for running requests any more. - The functional test counts a keep-mode pause and a resume per sync. Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on top of this PR, before the keep-mode pause (drained job 3069784 vs serve-through job 3074960): - post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s - max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error 0.00064 -> 0.00072 (gates 1.05 / 0.002) - steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is generation-bound, so the drain overlapped the trainer's wait for rollouts (exposed_generation ~0 -> ~71 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
vLLM's wait pause drains only running requests and never starts a waiting one, so a decode request still receiving its prompt KV keeps its blocks and the cache reset that completes the pause fails. Found on GB200 (tyche): the second refit of the P/D SingleController recipe failed in reset_prefix_cache. After every prefill engine has drained, poll each decode engine's vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps up with 512 rollouts per step. The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in place, but vLLM's cache reset raises while a decode engine still waits for remote KV or a prefill engine still holds blocks it handed off. So a disaggregated fleet drains in this order when the pause clears caches (recompute_kv_cache_after_weight_updates, the default), and freezes in the same order otherwise. control_timeout_s must then cover request_timeout_s, since it bounds the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Summary
Implementing the same feature as #3839 for the single controller path.
recompute_kv_cache_after_weight_updateseffective for active vLLM requests by clearing their KV state during native pause; the previous post-refit prefix-cache reset could leave running-request KV untouched while reporting successBehavior change
This PR also enables KV-cache recomputation by default, beyond the SingleController pause/resume lifecycle:
grpo.async_grpo.recompute_kv_cache_after_weight_updates(legacy async GRPO),ppo.async_ppo.recompute_kv_cache_after_weight_updates(legacy async PPO), andasync_rl.recompute_kv_cache_after_weight_updates(SingleController) now default totrue. For vLLM's native async path this clears paused requests' KV, repeating prefill after refit; it can reduce throughput.false. For legacy async GRPO with Megatron generation and for SingleController, pairtruewithpolicy.generation.mcore_generation_config.kv_cache_management_mode: recompute, orfalsewithpersist/offload. Setup rejects inconsistent pairs.kv_cache_management_mode: recomputeinstead ofpersist; explicitly selectingpersistoroffloadremains valid.falsepropagates the opt-out.true/recompute.Implementation
SingleControllerActor._sync_weights()continues to close the rollout-dispatch gate and cancel sampler-stale native requests first. After refit membership reconciliation and optional FP8 KV-scale calibration, it asks the generation backend to pause throughpause_generation_for_refit(clear_cache=...), runs the existingWeightSynchronizerrefit (including its one-shot recovery retry), and callsresume_generation_after_refit()before reopening dispatch.SingleController passes the existing refit timeout-plus-grace budget to pause and resume, with a 360-second fallback when
refit_timeout_s=None. The collective's existing disabled-watchdog behavior and_sync_weights_withinhelper/docstring remain unchanged. Pause and refit failures enter the existing one-shot fleet recovery, but survivors are marked as holding partial weights only if a transfer actually started. After rebuilding membership, the controller pauses every retry participant, including restarted engines, before transferring weights again.Resume waits for all leader RPCs under one shared timeout. With fleet health enabled, confirmed
ActorDiedErrorfailures remove only the dead engines; dispatch reopens if enough survivors successfully resumed. A late error from an old actor does not condemn its replacement. Timeouts, temporary actor unavailability, false acknowledgements, and exhausted fleets still stop the run with dispatch closed. Resume never retries the weight transfer.vLLM preserves active request state with native keep-mode pause. With
clear_cache=True, vLLM clears the KV state owned by those paused requests so they recompute it after refit. This fixes a pre-existing correctness gap: the former asyncreset_prefix_cachepath did not clear KV blocks held by running requests, and itsNoneworker results could still be reduced to success. This PR does not change theNone-reduction ininvalidate_kv_cache; the native async pause path instead no longer calls that function. Backends without native pause keep their existing in-flight refit behavior and receive the shared warning; when recompute is requested, they must confirm equivalent engine-side invalidation or the controller raisesNotImplementedErrorinstead of silently claiming recomputation.The lifecycle is in the shared SingleController
_sync_weights()path, so it applies to GRPO and PPO and to native and NeMo-Gym rollouts.Historical performance snapshot
For the previously linked Llama 3.1 8B SingleController runs at
e16a857503c0843df20d3b58f737be6bcde6bf76(not the current head), the mediantiming/train/total_step_timeover optimizer steps 1–10 was:Recompute was 3.16% slower by this median. This includes step 1 and is one historical pair, not a steady-state estimate, statistical result, or current-head performance claim. These runs did not record generation-tokens/s, so that requested metric cannot be supplied from their history. The logged valid-token throughput includes training behavior and must not be relabeled as generation throughput.
Validation
Current review fixes (
d001be04e610f3a1b2d5c5900606cb8d7fde1114):--noconftest, not the repository GPU test environment. Existing dependency/marker/timeout-thread warnings and mock-object shutdown messages remain; no GPU engine or distributed training runtime is validated by these CPU checks.git diff --checkpassed.Historical validation
The following evidence is tied to its stated revisions and does not establish current-head CI, performance, or GPU recovery readiness.
Review follow-up (
70f3b3967c8026863b4ffe4bd108a5803f7a1d45):refit_timeout_s=null.cbf5d3be97d3e64f4d2a03585c61a0137d9f67adpassed: 63 successful jobs, 9 skipped.cbf5d3be97d3e64f4d2a03585c61a0137d9f67adagainst main9c8778b1404e84c64bc73f671427234ba304e41c:nccl_reshardstep-boundary kill passed 24/24 on both;ncclstep-boundary recovery failed with the same rebuild traceback on both. PR-only kill-during-refit and restart variants also failed with that rebuild error. These are the reviewer's reported results, not independently rerun here. They do not replace the unresolved historical manual GPU evidence or constitute a complete nine-variant suite on the follow-up SHA.Review fixes (
bad0cb1691e9e20f0130219ddaee22b76a7a07b4):git diff --checkpass.RefitAbortedwithin its deadline. The controller/backend methods were source-extracted; no vLLM engine or GPU was exercised. The first attempt stopped during automatic uv workspace packaging; disabling that local-only packaging allowed the test to run.Previous merge (
d8e713dbbf6ba3d08f481c53d25e0a3e9fff41f1):mainatfeff70f5b6239d90dc703ae79b973ea809550732, including feat(sc): restart dead generation shards and re-admit them at the next refit #3592. Preserved participant-aware refit completion and shard re-admission. Recovery retries now re-pause the rebuilt fleet, covering both returning shards and replacement engines that reuse a shard index.18608218failed during environment setup (/opt/nemo_rl_venv/bin/pythonwas absent from the selected image), before pytest started. It provides no runtime validation.Earlier validation:
git diff --checkpassed17015948innemo_rl.0825.sqsh: 92 focused tests passedtest_single_controller_actor.pytest_refit_recovery.py2134in the same image/account ran the fulltest_checkpointing.py: 62 passedgen_handle=None, CW-DFW Slurm job17510464in the same image/account ran the fulltest_train_pump_e2e.pyunder--hf-gated --mcore-onlywith a forced worker-environment rebuild: 1 passed, 66 warnings in 123.19s; SlurmCOMPLETED/0:0in 6:27e16a857503c0843df20d3b58f737be6bcde6bf76. These runs predate the subsequent merges frommainand do not validate the current PR head.grpo-llama3.1-8b-instruct-2n8g-async-1off-single-controller-streaming2.yamlworkload: 2 nodes / 16 H100 GPUs, 64 prompts x 32 generations, global batch size 2048, max sequence length 409617016712,COMPLETED/0:0, 10 optimizer steps and 11 successful pause/refit/resume cycles; W&B17016711,COMPLETED/0:0, 10 optimizer steps and 11 successful pause/refit/resume cycles; W&B