Skip to content

feat: pause SingleController refits and enable KV recomputation by default - #3929

Merged
terrykong merged 26 commits into
mainfrom
youngeunk/single-controller-refit-pause
Oct 19, 2026
Merged

terrykong merged 26 commits into
mainfrom
youngeunk/single-controller-refit-pause

Conversation

@youngeunkwon0405

@youngeunkwon0405 youngeunkwon0405 commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implementing the same feature as #3839 for the single controller path.

  • apply the common generation pause/resume lifecycle around SingleController weight refits
  • make recompute_kv_cache_after_weight_updates effective for active vLLM requests by clearing their KV state during native pause; the previous post-refit prefix-cache reset could leave running-request KV untouched while reporting success
  • keep rollout dispatch closed on unrecovered lifecycle failures; tolerate confirmed resume-time actor deaths when enough healthy engines survive
  • make vLLM pause/resume target the surviving refit leaders after fleet membership recovery
  • enable KV recomputation by default, with explicit backend-aware opt-outs and compatibility fixes

Behavior change

This PR also enables KV-cache recomputation by default, beyond the SingleController pause/resume lifecycle:

  • grpo.async_grpo.recompute_kv_cache_after_weight_updates (legacy async GRPO), ppo.async_ppo.recompute_kv_cache_after_weight_updates (legacy async PPO), and async_rl.recompute_kv_cache_after_weight_updates (SingleController) now default to true. For vLLM's native async path this clears paused requests' KV, repeating prefill after refit; it can reduce throughput.
  • Opt out with the corresponding algorithm-specific flag set explicitly to false. For legacy async GRPO with Megatron generation and for SingleController, pair true with policy.generation.mcore_generation_config.kv_cache_management_mode: recompute, or false with persist/offload. Setup rejects inconsistent pairs.
  • Synchronous legacy GRPO does not consume the async recompute flag and is exempt from that consistency check. However, synchronous Megatron recipes/nightlies that inherit the shared exemplar now inherit kv_cache_management_mode: recompute instead of persist; explicitly selecting persist or offload remains valid.
  • The TRT-LLM exemplar's generation recompute setting interpolates the legacy GRPO flag, so its inherited default changes too; explicitly setting the GRPO flag to false propagates the opt-out.
  • SGLang's existing weight synchronizer always pauses and flushes KV before transferring weights, even with the flag off. Its paired lifecycle acknowledgements prevent a second cache flush after generation resumes; they do not add another pause/resume RPC or timeout.
  • The legacy async Omni video recipe explicitly keeps recomputation off to preserve its existing persist-mode/CUDA-graph and long-video-prefill behavior. The SingleController CLEVR recipe remains true/recompute.

Implementation

SingleControllerActor._sync_weights() continues to close the rollout-dispatch gate and cancel sampler-stale native requests first. After refit membership reconciliation and optional FP8 KV-scale calibration, it asks the generation backend to pause through pause_generation_for_refit(clear_cache=...), runs the existing WeightSynchronizer refit (including its one-shot recovery retry), and calls resume_generation_after_refit() before reopening dispatch.

SingleController passes the existing refit timeout-plus-grace budget to pause and resume, with a 360-second fallback when refit_timeout_s=None. The collective's existing disabled-watchdog behavior and _sync_weights_within helper/docstring remain unchanged. Pause and refit failures enter the existing one-shot fleet recovery, but survivors are marked as holding partial weights only if a transfer actually started. After rebuilding membership, the controller pauses every retry participant, including restarted engines, before transferring weights again.

Resume waits for all leader RPCs under one shared timeout. With fleet health enabled, confirmed ActorDiedError failures remove only the dead engines; dispatch reopens if enough survivors successfully resumed. A late error from an old actor does not condemn its replacement. Timeouts, temporary actor unavailability, false acknowledgements, and exhausted fleets still stop the run with dispatch closed. Resume never retries the weight transfer.

vLLM preserves active request state with native keep-mode pause. With clear_cache=True, vLLM clears the KV state owned by those paused requests so they recompute it after refit. This fixes a pre-existing correctness gap: the former async reset_prefix_cache path did not clear KV blocks held by running requests, and its None worker results could still be reduced to success. This PR does not change the None-reduction in invalidate_kv_cache; the native async pause path instead no longer calls that function. Backends without native pause keep their existing in-flight refit behavior and receive the shared warning; when recompute is requested, they must confirm equivalent engine-side invalidation or the controller raises NotImplementedError instead of silently claiming recomputation.

The lifecycle is in the shared SingleController _sync_weights() path, so it applies to GRPO and PPO and to native and NeMo-Gym rollouts.

Historical performance snapshot

For the previously linked Llama 3.1 8B SingleController runs at e16a857503c0843df20d3b58f737be6bcde6bf76 (not the current head), the median timing/train/total_step_time over optimizer steps 1–10 was:

KV mode Median step time
Preserve 58.43 s
Recompute 60.28 s

Recompute was 3.16% slower by this median. This includes step 1 and is one historical pair, not a steady-state estimate, statistical result, or current-head performance claim. These runs did not record generation-tokens/s, so that requested metric cannot be supplied from their history. The logged valid-token throughput includes training behavior and must not be relabeled as generation throughput.

Validation

Current review fixes (d001be04e610f3a1b2d5c5900606cb8d7fde1114):

  • Addressed the checkpointing fake-backend failure, the equivalent training-pump fixture failure in L0, the legacy async video recipe mismatch, SGLang's duplicate post-resume flush, sync-GRPO overvalidation, the stale MCore default comment, and launcher coverage. vLLM resume now uses one snapshot of shard/worker/leader triples; this removes duplicated mapping logic rather than claiming a demonstrated current mapping race.
  • 363 selected full-module CPU tests passed locally in two runs (309 + 54) on macOS/Python 3.13.14 with Torch 2.13.0 and Ray 2.58.0. This includes the full checkpointing, controller, refit-recovery, and refit-resume modules; the 17 Megatron launcher combinations; 15 setup-level flag/mode cases; and the affected native vLLM lifecycle tests. Tests used an isolated dependency environment and --noconftest, not the repository GPU test environment. Existing dependency/marker/timeout-thread warnings and mock-object shutdown messages remain; no GPU engine or distributed training runtime is validated by these CPU checks.
  • Ruff 0.9.9 lint/format, recipe minimization, and git diff --check passed.
  • Exact-head CI has been requested for this commit; results are pending. Do not treat historical green runs below as validation of this head.
  • The requested H100 Megatron nightly performance check and full-mode SingleController/router lane are being prepared but have not run on this head. Historical W&B runs do not establish current-head performance.

Historical validation

The following evidence is tied to its stated revisions and does not establish current-head CI, performance, or GPU recovery readiness.

Review follow-up (70f3b3967c8026863b4ffe4bd108a5803f7a1d45):

  • Docs and tests only; no production behavior changes. Restored non-colocated timeout-sizing guidance, including separate lifecycle budgets, recovery/retry time, and the unbounded collective wait when refit_timeout_s=null.
  • Added a pause-timeout/no-absent-shard regression and covered confirmed actor death during the retry pause. 64 focused source-extracted control-flow tests pass locally on macOS/Python 3.12; this is not full-module or GPU validation. Existing daemon-thread/event-loop-close warnings remain. An in-memory mutation restoring the old pause classification makes the new healthy-survivor assertion fail.
  • Ruff 0.9.9 lint/format, Python 3.13.14 syntax, and diff checks pass. CI has been requested for the full follow-up SHA; results are pending.
  • CI on preceding head cbf5d3be97d3e64f4d2a03585c61a0137d9f67ad passed: 63 successful jobs, 9 skipped.
  • Separate reviewer-reported fault-injection results are pinned to cbf5d3be97d3e64f4d2a03585c61a0137d9f67ad against main 9c8778b1404e84c64bc73f671427234ba304e41c: nccl_reshard step-boundary kill passed 24/24 on both; nccl step-boundary recovery failed with the same rebuild traceback on both. PR-only kill-during-refit and restart variants also failed with that rebuild error. These are the reviewer's reported results, not independently rerun here. They do not replace the unresolved historical manual GPU evidence or constitute a complete nine-variant suite on the follow-up SHA.

Review fixes (bad0cb1691e9e20f0130219ddaee22b76a7a07b4):

  • Addressed initial-pause failure classification, confirmed actor death during resume, bounded lifecycle calls when the collective watchdog is disabled, and synchronous-vLLM unsupported-hook behavior.
  • Ruff 0.9.9 lint/format and git diff --check pass.
  • GitHub CI for this head has been triggered and is queued.
  • 60 focused tests pass with source-extracted controller/backend methods and the real fleet-health state machine on macOS/Python 3.11, using Ray 2.56.1 (the lockfile version at that historical revision; the current lock pins Ray 2.58.0 and vLLM 0.29.0). This bypasses the full import stack and is not full-runtime or GPU validation. Two existing daemon-thread/event-loop-close warnings remain in deadline tests.
  • A supplemental CPU-only check with real Ray 2.56.1 actors passed: an intentionally dying leader was recorded dead, a delayed survivor acknowledged resume before success, and a separate pending resume raised RefitAborted within its deadline. The controller/backend methods were source-extracted; no vLLM engine or GPU was exercised. The first attempt stopped during automatic uv workspace packaging; disabling that local-only packaging allowed the test to run.
  • The requested nine-variant GPU fleet-health fault-injection suite has not run on this head. The historical GPU runs below do not validate these changes.

Previous merge (d8e713dbbf6ba3d08f481c53d25e0a3e9fff41f1):

  • Merged main at feff70f5b6239d90dc703ae79b973ea809550732, including feat(sc): restart dead generation shards and re-admit them at the next refit #3592. Preserved participant-aware refit completion and shard re-admission. Recovery retries now re-pause the rebuilt fleet, covering both returning shards and replacement engines that reuse a shard index.
  • Ruff 0.9.9 lint/format, Python syntax, and diff checks pass. The existing bounded refit helper and docstring are unchanged.
  • 46 isolated recovery control-flow tests passed locally using source-extracted controller methods and Ray 2.55.1 on macOS/Python 3.11. This bypasses the full GPU import stack and is not full-module or GPU validation. The run reported two daemon-thread event-loop-close warnings in existing deadline tests and one temporary pytest-cache warning.
  • Both new restart/retry regression cases fail when the retry pause is removed in memory, and pass with the fix.
  • GitHub CI for the preceding merged head passed. Its fast functional lane does not cover the multi-shard fleet-health fault-injection variants.
  • Previous-head CW-DFW job 18608218 failed during environment setup (/opt/nemo_rl_venv/bin/python was absent from the selected image), before pytest started. It provides no runtime validation.

Earlier validation:

  • Ruff 0.9.9 check and format verification passed for all modified Python files
  • git diff --check passed
  • CW-DFW Slurm job 17015948 in nemo_rl.0825.sqsh: 92 focused tests passed
    • full test_single_controller_actor.py
    • full test_refit_recovery.py
    • native vLLM worker and generation pause/resume tests
  • after L0 exposed a checkpointing fixture without the new lifecycle hooks, CW-DFW Slurm job 2134 in the same image/account ran the full test_checkpointing.py: 62 passed
  • after Megatron L0 exposed a training-pump fixture with gen_handle=None, CW-DFW Slurm job 17510464 in the same image/account ran the full test_train_pump_e2e.py under --hf-gated --mcore-only with a forced worker-environment rebuild: 1 passed, 66 warnings in 123.19s; Slurm COMPLETED / 0:0 in 6:27
  • tests cover native and NeMo-Gym selection, recompute on/off, supported and unsupported pause, resume/refit failure gating, and keeping generation paused until a recovery retry completes
  • earlier full-size real-GPU SingleController GRPO validation on CW-DFW at commit e16a857503c0843df20d3b58f737be6bcde6bf76. These runs predate the subsequent merges from main and do not validate the current PR head.
    • unchanged grpo-llama3.1-8b-instruct-2n8g-async-1off-single-controller-streaming2.yaml workload: 2 nodes / 16 H100 GPUs, 64 prompts x 32 generations, global batch size 2048, max sequence length 4096
    • only runtime overrides were 10 steps, recompute mode, W&B destinations, and vLLM generation metrics logging
    • KV-cache preserve: Slurm 17016712, COMPLETED / 0:0, 10 optimizer steps and 11 successful pause/refit/resume cycles; W&B
    • KV-cache recompute: Slurm 17016711, COMPLETED / 0:0, 10 optimizer steps and 11 successful pause/refit/resume cycles; W&B
    • the initial sync had zero in-flight groups; every one of the 10 training refits paused and preserved nonzero in-flight groups in both runs

@youngeunkwon0405
youngeunkwon0405 requested review from a team as code owners August 31, 2026 08:29
@copy-pr-bot

copy-pr-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Aug 31, 2026
@youngeunkwon0405 youngeunkwon0405 added the CI:L2 Run doctests, unit tests, functional tests, and convergence tests label Aug 31, 2026
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test e16a857

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

Real-GPU validation completed at exact PR head e16a857503c0843df20d3b58f737be6bcde6bf76.

I ran the full-size native SingleController vLLM GRPO recipe (grpo-llama3.1-8b-instruct-2n8g-async-1off-single-controller-streaming2.yaml) on CW-DFW with the requested coreai_dlalgo_genai account and nemo_rl.0825.sqsh image. The workload dimensions were unchanged: 2 nodes / 16 H100 GPUs, 64 prompts per step x 32 generations, global batch size 2048, and max sequence length 4096. The only runtime overrides were 10 steps, recompute mode, logging destinations, and enable_vllm_metrics_logger=true.

  • recompute_kv_cache_after_weight_updates=false: Slurm 17016712, COMPLETED / 0:0 in 17:55; 10 optimizer steps; 11 pause/refit/resume cycles (initial sync + 10 training refits); all 10 training refits paused nonzero in-flight groups. W&B run
  • recompute_kv_cache_after_weight_updates=true: Slurm 17016711, COMPLETED / 0:0 in 18:14; 10 optimizer steps; 11 pause/refit/resume cycles (initial sync + 10 training refits); all 10 training refits paused nonzero in-flight groups. W&B run

The only traceback in each log was a post-success W&B service BrokenPipeError during Ray process teardown; both Slurm jobs exited successfully after W&B had emitted the final run summary.

@youngeunkwon0405
youngeunkwon0405 force-pushed the youngeunk/single-controller-refit-pause branch from e16a857 to daa045d Compare August 31, 2026 10:00
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test daa045d

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

Addressed the L0_Unit_Tests_Other failure at new head daa045de89a5a0071a1b09cc4b50adcf262a1534.

The failure was PR-caused but test-only: TestCounterRestore.test_resumed_pump_continues_to_max_steps used gen_handle=object() and now reaches the real _sync_weights() lifecycle, so the fixture lacked pause_generation_for_refit() / resume_generation_after_refit(). I replaced it with an unsupported-backend fake implementing the common lifecycle contract; no production code changed from the GPU-validated head.

Validation:

  • pinned Ruff 0.9.9 check and format check passed
  • CW-DFW Slurm 2134, nemo_rl.0825.sqsh, coreai_dlalgo_genai: COMPLETED / 0:0; full tests/unit/single_controller/test_checkpointing.py: 62 passed, 17 warnings in 20.93s
  • exact-head GitHub CI was reauthorized with /ok to test daa045de89a5a0071a1b09cc4b50adcf262a1534

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405
youngeunkwon0405 force-pushed the youngeunk/single-controller-refit-pause branch from daa045d to 15074e4 Compare August 31, 2026 14:50
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

Addressed the remaining L0_Unit_Tests_Megatron failure at new head 15074e4ccfe626a71df6972d247e6bf394f77eef.

The failure was PR-caused but test-only: test_train_pump_drives_mcore_training_step passed gen_handle=None and now reaches the real _sync_weights() lifecycle, so the fixture lacked the common pause/resume hooks. I replaced it with an unsupported-backend fake implementing pause_generation_for_refit() and resume_generation_after_refit(). No production code changed from the two 10-step GPU-validated runs.

Validation:

  • pinned Ruff 0.9.9 check and format verification passed; git diff --check passed
  • CW-DFW Slurm 17510464, nemo_rl.0825.sqsh, coreai_dlalgo_genai: COMPLETED / 0:0 in 6:27
  • full tests/unit/single_controller/test_train_pump_e2e.py under --hf-gated --mcore-only: 1 passed, 66 warnings in 123.19s

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 15074e4

@youngeunkwon0405 youngeunkwon0405 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Team review: 4 agents (rl-expert, test-agent, bug-finder, devil's advocate) analyzed this PR; every finding below was independently re-verified against source, including the pinned vLLM 0.25.1 wheel.

Verified clean (no action): clear_cache-at-pause genuinely subsumes post-refit invalidation (vLLM keep-mode pause_scheduler resets caches at pause); _refit_leader_workers() targets exactly the set the old run_rank_0_only_axes dispatch selected (incl. pp>1) and matches the existing refit dispatches; interface default hooks keep unsupported backends safe; fail-stop resume semantics mirror trajectory_collector; removing the drain-gate TODO is consistent (docs + issue #2625 still track it). Ruff 0.9.9 check+format clean. The A/B W&B evidence (preserve vs recompute, 11 pause/refit/resume cycles each) is solid.

Findings: 1 hardening suggestion, 1 docs note, 2 optional test suggestions (both verified passing locally on CPU) — all inline.

Generated by Claude Code

Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread docs/guides/single-controller.md Resolved
@youngeunkwon0405 youngeunkwon0405 self-assigned this Aug 31, 2026
…roller-refit-pause

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 1f68285

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

Post-review validation for exact head 1f68285:

  • Local: Ruff 0.9.9 check and format check, compileall, and git diff --check passed.
  • Focused CW-DFW GPU tests: Slurm 17531796, 1 node / 8 H100, COMPLETED 0:0. SingleController recovery/lifecycle suites: 92 passed. Focused vLLM native-pause/cache-result cases: 8 passed.
  • Real GRPO validation: Slurm 17531931, 2 nodes / 16 H100, account coreai_dlalgo_genai, full grpo-llama3.1-8b-instruct-2n8g-async-1off-single-controller-streaming2 recipe, COMPLETED 0:0 in 19m30s.
    • 10/10 steps completed; 64 prompts x 32 generations, global batch 2048, max generation/total sequence length 4096. No batch-size or sequence-length reduction.
    • async_rl.recompute_kv_cache_after_weight_updates=true.
    • vLLM generation metrics logging enabled.
    • Observed 11 successful pause/refit/resume cycles (initial sync plus 10 optimizer steps), 11 successful refits, and prefix-cache reset logs on every cycle. The 10 training-cycle pauses preserved 58-64 in-flight rollout groups.
    • No RefitAborted or NotImplementedError; final receipt: SC run complete with train_steps=10 and trainer_version=10.
  • W&B: https://wandb.ai/nvidia/async-grpo-pr-validation/runs/gabgq6rm

Claude review: all four threads were addressed in 1f68285, replied to, and resolved: bounded lifecycle calls, timeout-budget documentation, recovery ordering coverage, and pause-failure coverage. Exact-head CI was retriggered; branch-up-to-date, lint, docs, and script-coverage checks are passing, with container builds still running.

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 463f247

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

Simplified the post-review implementation at exact head 463f247737b65a7257a5f3dfaec94b1b3881cdbf.

  • Removed the generalized _run_refit_call_within abstraction and restored the existing specialized _sync_weights_within implementation and its docstring unchanged.
  • Pause, resume, and fallback cache invalidation again use direct asyncio.to_thread calls in _sync_weights().
  • Removed the unrelated vLLM prefix-reset result plumbing and its tests.
  • Kept the correctness guard: when native pause is unavailable and recomputation is requested, the backend must confirm equivalent active-request KV handling or the run raises NotImplementedError instead of silently preserving stale KV state.
  • The simplification follow-up removed 139 lines and added 43.

Validation at this exact head:

  • Ruff 0.9.9 check and format check, Python compile, and git diff --check: passed.
  • Focused CW-DFW GPU tests: Slurm 17534872, 1 node / 8 H100, coreai_dlalgo_genai, COMPLETED 0:0 in 2m38s. SingleController suites: 91 passed; focused vLLM native-pause lifecycle cases: 3 passed.
  • Real GRPO validation: Slurm 17535168, 2 nodes / 16 H100, coreai_dlalgo_genai, full grpo-llama3.1-8b-instruct-2n8g-async-1off-single-controller-streaming2 recipe, COMPLETED 0:0 in 19m10s.
    • 10/10 optimizer steps completed; final receipt: SC run complete: {'train_steps': 10, 'trainer_version': 10}.
    • No batch-size or sequence-length reduction: 64 prompts x 32 generations, global batch 2048, max generation/total sequence length 4096.
    • async_rl.recompute_kv_cache_after_weight_updates=true and vLLM generation metrics logging were enabled.
    • Observed 11 successful pause/refit/resume cycles (initial sync plus 10 training refits). Every training refit paused nonzero in-flight work (42-64 rollout groups).
    • No RefitAborted or NotImplementedError.
    • W&B run

The only post-success traceback was the same W&B service BrokenPipeError during Ray teardown seen in prior successful validations; it occurred after the final W&B summary and SC run complete, and the Slurm job exited successfully.

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 463f247

@youngeunkwon0405 youngeunkwon0405 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L2 Run doctests, unit tests, functional tests, and convergence tests labels Sep 1, 2026
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test df75b8c

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test d001be0

@youngeunkwon0405 youngeunkwon0405 changed the title feat(single-controller): pause generation during refit feat: pause SingleController refits and enable KV recomputation by default Oct 17, 2026
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 9c15e56

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 064a49e

@youngeunkwon0405 youngeunkwon0405 added CI:L1 Run doctests, unit tests, and functional tests and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels Oct 19, 2026
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 064a49e

@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 064a49e

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405

Copy link
Copy Markdown
Contributor Author

/ok to test 61dfb29

@youngeunkwon0405

youngeunkwon0405 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

H100 Megatron nightly results on 61dfb2923018b14f5f22eaa51891b2d632305ddf:

The original grpo-llama3.2-1b-instruct-1n8g-megatron_generation workload completed all 500 training steps on eight H100 GPUs using nemo_rl.1018.sqsh. All four original metric gates passed:

Check Observed Required
Median token probability error 1.00912154 <1.1
Step-500 token probability error 1.00711179 <1.1
Step-500 reward 0.25 >0.1
Median step time 10.027695s <12s

The median includes all 500 steps, including the 55.0465s first step. Each required metric has exactly 500 finite values for steps 1–500. The W&B run is finished and independently contains all 500 steps.

Post-processing qualification: the original training allocation ended FAILED (exit 1:0) after training and metrics export, because the isolated uv checker environment could not resolve rich offline. The saved metrics were subsequently checked with the unchanged original checker and thresholds, using the same image's preinstalled Python environment in a separate successful CPU-only post-processing job. The original failed batch record is retained; this is a recovered metrics-validation result, not a claim that the original launcher exited successfully.

Router validation: the full-mode L1_Functional_Tests_SingleController_2 lane passed on both H100 and GB200. Each actually executed the 10-step Gym case with the generation router and fleet health enabled, and passed the original median(train/gen_kl_error) < 1.3 and max(train/reward) > 0 assertions.

These two-GPU runs validate router pass-through, not failover/recovery: scenarios requiring at least three GPUs were explicitly skipped.

Update: overall L1 CI, attempt 2 completed SUCCESS on the same PR head. The previously failing H100 Megatron-generation, GB200 SFT, and GB200 SingleController/Gym lanes all passed on retry without source changes. The CI-quality gate also passed. The original failures remain preserved; retry success does not establish the exact cause of each original failure.

@terrykong
terrykong merged commit 7344ae7 into main Oct 19, 2026
209 of 213 checks passed
@terrykong
terrykong deleted the youngeunk/single-controller-refit-pause branch October 19, 2026 20:45
jthomson04 added a commit that referenced this pull request Oct 19, 2026
RL#4719 implemented the SingleController's refit pause for Dynamo as a
drain, because Dynamo then reloaded weights layerwise through its own
path. Since the refit moved into NeMo RL's vLLM worker extension, each
refit is one update_weights_from_distributed call per engine, the same
update the vLLM backend takes. The drain held every refit until each
running request finished.

- DynamoGeneration.pause_generation_for_refit now freezes every engine
  through Dynamo's pause_generation route with vLLM's keep mode, as the
  vLLM backend pauses its own engines (#3929): frozen requests continue
  on the new weights after the resume, and clear_cache
  (recompute_kv_cache_after_weight_updates, the default) drops their KV
  so they recompute it. timeout_s bounds each engine's pause and resume.
- flush_cache keeps its drain for the legacy loop's post-refit
  invalidation.
- Drop the control_timeout_s >= request_timeout_s check: nothing on the
  SingleController path waits for running requests any more.
- The functional test counts a keep-mode pause and a resume per sync.

Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on
top of this PR, before the keep-mode pause (drained job 3069784 vs
serve-through job 3074960):
- post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s
- max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error
  0.00064 -> 0.00072 (gates 1.05 / 0.002)
- steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is
  generation-bound, so the drain overlapped the trainer's wait for
  rollouts (exposed_generation ~0 -> ~71 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 19, 2026
vLLM's wait pause drains only running requests and never starts a waiting
one, so a decode request still receiving its prompt KV keeps its blocks and
the cache reset that completes the pause fails. Found on GB200 (tyche): the
second refit of the P/D SingleController recipe failed in reset_prefix_cache.

After every prefill engine has drained, poll each decode engine's
vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then
pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps
up with 512 rollouts per step.

The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in
place, but vLLM's cache reset raises while a decode engine still waits for
remote KV or a prefill engine still holds blocks it handed off. So a
disaggregated fleet drains in this order when the pause clears caches
(recompute_kv_cache_after_weight_updates, the default), and freezes in the
same order otherwise. control_timeout_s must then cover request_timeout_s,
since it bounds the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 20, 2026
RL#4719 implemented the SingleController's refit pause for Dynamo as a
drain, because Dynamo then reloaded weights layerwise through its own
path. Since the refit moved into NeMo RL's vLLM worker extension, each
refit is one update_weights_from_distributed call per engine, the same
update the vLLM backend takes. The drain held every refit until each
running request finished.

- DynamoGeneration.pause_generation_for_refit now freezes every engine
  through Dynamo's pause_generation route with vLLM's keep mode, as the
  vLLM backend pauses its own engines (NVIDIA-NeMo#3929): frozen requests continue
  on the new weights after the resume, and clear_cache
  (recompute_kv_cache_after_weight_updates, the default) drops their KV
  so they recompute it. timeout_s bounds each engine's pause and resume.
- flush_cache keeps its drain for the legacy loop's post-refit
  invalidation.
- Drop the control_timeout_s >= request_timeout_s check: nothing on the
  SingleController path waits for running requests any more.
- The functional test counts a keep-mode pause and a resume per sync.

Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on
top of this PR, before the keep-mode pause (drained job 3069784 vs
serve-through job 3074960):
- post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s
- max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error
  0.00064 -> 0.00072 (gates 1.05 / 0.002)
- steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is
  generation-bound, so the drain overlapped the trainer's wait for
  rollouts (exposed_generation ~0 -> ~71 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 20, 2026
vLLM's wait pause drains only running requests and never starts a waiting
one, so a decode request still receiving its prompt KV keeps its blocks and
the cache reset that completes the pause fails. Found on GB200 (tyche): the
second refit of the P/D SingleController recipe failed in reset_prefix_cache.

After every prefill engine has drained, poll each decode engine's
vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then
pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps
up with 512 rollouts per step.

The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in
place, but vLLM's cache reset raises while a decode engine still waits for
remote KV or a prefill engine still holds blocks it handed off. So a
disaggregated fleet drains in this order when the pause clears caches
(recompute_kv_cache_after_weight_updates, the default), and freezes in the
same order otherwise. control_timeout_s must then cover request_timeout_s,
since it bounds the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 20, 2026
RL#4719 implemented the SingleController's refit pause for Dynamo as a
drain, because Dynamo then reloaded weights layerwise through its own
path. Since the refit moved into NeMo RL's vLLM worker extension, each
refit is one update_weights_from_distributed call per engine, the same
update the vLLM backend takes. The drain held every refit until each
running request finished.

- DynamoGeneration.pause_generation_for_refit now freezes every engine
  through Dynamo's pause_generation route with vLLM's keep mode, as the
  vLLM backend pauses its own engines (NVIDIA-NeMo#3929): frozen requests continue
  on the new weights after the resume, and clear_cache
  (recompute_kv_cache_after_weight_updates, the default) drops their KV
  so they recompute it. timeout_s bounds each engine's pause and resume.
- flush_cache keeps its drain for the legacy loop's post-refit
  invalidation.
- Drop the control_timeout_s >= request_timeout_s check: nothing on the
  SingleController path waits for running requests any more.
- The functional test counts a keep-mode pause and a resume per sync.

Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on
top of this PR, before the keep-mode pause (drained job 3069784 vs
serve-through job 3074960):
- post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s
- max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error
  0.00064 -> 0.00072 (gates 1.05 / 0.002)
- steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is
  generation-bound, so the drain overlapped the trainer's wait for
  rollouts (exposed_generation ~0 -> ~71 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 20, 2026
vLLM's wait pause drains only running requests and never starts a waiting
one, so a decode request still receiving its prompt KV keeps its blocks and
the cache reset that completes the pause fails. Found on GB200 (tyche): the
second refit of the P/D SingleController recipe failed in reset_prefix_cache.

After every prefill engine has drained, poll each decode engine's
vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then
pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps
up with 512 rollouts per step.

The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in
place, but vLLM's cache reset raises while a decode engine still waits for
remote KV or a prefill engine still holds blocks it handed off. So a
disaggregated fleet drains in this order when the pause clears caches
(recompute_kv_cache_after_weight_updates, the default), and freezes in the
same order otherwise. control_timeout_s must then cover request_timeout_s,
since it bounds the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 23, 2026
RL#4719 implemented the SingleController's refit pause for Dynamo as a
drain, because Dynamo then reloaded weights layerwise through its own
path. Since the refit moved into NeMo RL's vLLM worker extension, each
refit is one update_weights_from_distributed call per engine, the same
update the vLLM backend takes. The drain held every refit until each
running request finished.

- DynamoGeneration.pause_generation_for_refit now freezes every engine
  through Dynamo's pause_generation route with vLLM's keep mode, as the
  vLLM backend pauses its own engines (NVIDIA-NeMo#3929): frozen requests continue
  on the new weights after the resume, and clear_cache
  (recompute_kv_cache_after_weight_updates, the default) drops their KV
  so they recompute it. timeout_s bounds each engine's pause and resume.
- flush_cache keeps its drain for the legacy loop's post-refit
  invalidation.
- Drop the control_timeout_s >= request_timeout_s check: nothing on the
  SingleController path waits for running requests any more.
- The functional test counts a keep-mode pause and a resume per sync.

Measured on tyche GB200, 1n4g, 8 steps, with the prefill/decode stack on
top of this PR, before the keep-mode pause (drained job 3069784 vs
serve-through job 3074960):
- post-step sync: 54.7-73.0 s -> 0.094-0.106 s; first sync 1.24 -> 1.19 s
- max token_mult_prob_error 1.016 -> 1.017, max gen_kl_error
  0.00064 -> 0.00072 (gates 1.05 / 0.002)
- steady-state step time (steps 2-8) 89.8 -> 90.0 s: that recipe is
  generation-bound, so the drain overlapped the trainer's wait for
  rollouts (exposed_generation ~0 -> ~71 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 23, 2026
vLLM's wait pause drains only running requests and never starts a waiting
one, so a decode request still receiving its prompt KV keeps its blocks and
the cache reset that completes the pause fails. Found on GB200 (tyche): the
second refit of the P/D SingleController recipe failed in reset_prefix_cache.

After every prefill engine has drained, poll each decode engine's
vllm:num_requests_waiting on its Dynamo /metrics until it reaches zero, then
pause decode. Raise the nightly's max_num_seqs so a single decode engine keeps
up with 512 rollouts per step.

The SingleController's refit pause (NVIDIA-NeMo#3929) freezes aggregated engines in
place, but vLLM's cache reset raises while a decode engine still waits for
remote KV or a prefill engine still holds blocks it handed off. So a
disaggregated fleet drains in this order when the pause clears caches
(recompute_kv_cache_after_weight_updates, the default), and freezes in the
same order otherwise. control_timeout_s must then cover request_timeout_s,
since it bounds the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants