Repository navigation
feat(single-controller): optionally run weight synchronizer in separate Actor - #5182
afarjallah-nv wants to merge 1 commit into
Conversation
…separate RefitActor Add async_rl.refit_actor.enabled (off by default). When enabled, the SingleController's weight synchronizer runs in its own CPU Ray actor on the controller's node, so a busy controller event loop no longer delays the refit transport and a slow refit no longer occupies a controller thread. Only the transport moves (sync_weights, init/reconcile_communicator). The rollout gate, generation pause/resume, refit membership decisions, the refit deadline and recovery stay in the controller. - RefitActor holds the synchronizer; RemoteWeightSynchronizer is the controller-side proxy. It re-ships the generation object when worker handles change, mirrors the refit membership the actor builds, and replaces a dead actor on the next is_stale / reconcile_communicator. - The proxy pickles as its actor handle only; the SingleControllerActor attaches it to its own generation plus a synchronizer factory. - Copies of the generation, policy and clusters inside the actor get a no-op shutdown so their __del__ safety nets cannot stop the real workers or kill the actor. - Rejected with colocated generation; NCCL transports only. - The controller awaits the actor's ObjectRef under the existing refit deadline; a timeout still raises RefitAborted into recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: afarjallah <afarjallah@nvidia.com>
saumishr
left a comment
There was a problem hiding this comment.
Reviewed the proxy's interface coverage, the dead-actor recovery path, and the replacement-synchronizer bootstrap against the source; all three are correct, so the findings below are mostly process and docs.
Two items that have no natural line to attach to:
[CI] This PR has no CI:* label. Every PR needs exactly one or the quality-check job stays red. CI:L0 fits ("new code"), or CI:L1 for the functional tier given this touches the refit path — the feature is off by default, so L0 is defensible.
[Question] Actor placement diverges from the nearest analog. RefitActor soft-pins to the controller's node to keep the RPC hop local, while AdvantageActor deliberately does the opposite and falls back to "SPREAD" with the comment "Ray's hybrid default packs a CPU-only actor onto the driver's node -- beside the controller it is relieving." Both rationales are reasonable for their workloads (I/O-bound fan-out vs. CPU-heavy tensor work), but since the sibling documents the opposite tradeoff, a sentence confirming the choice would help.
Also minor: is_stale now makes two blocking ray.get calls (ping + status) and is read synchronously inside the async _run_pumps at single_controller.py:883. It is startup-only so it costs nothing in steady state, but it is a blocking call on the event loop in a PR about not blocking the controller.
Things I expected to be problems and confirmed are fine, so they don't get re-litigated later: the proxy implements the complete WeightSynchronizer surface, so nothing falls through to the ABC; _generation and _built_membership do exist on both NCCL transports; dead-actor recovery works because RayActorError is already in the recovery except at line 4934 and reconcile_communicator replaces the actor before forwarding; a replacement synchronizer is never init_communicator'd, but reconcile_communicator(force=True) handles _built_membership is None by assuming the full fleet and rebuilding, and a fresh CollectiveWeightSynchronizer starts _stale = True; _build_refit_synchronizer matches both create_weight_synchronizer call sites argument for argument; and the max_concurrency=4 race between a stuck sync_weights and a recovery reconcile_communicator is the same race the in-process path already has with its orphaned thread. pyrefly.toml, the # pragma: no cover, copyright headers and sign-off are all in order, and the two edited exemplars have no counterpart in tests/unit/reference_configs/, so test_config_v2.py needs no update.
| return self | ||
|
|
||
|
|
||
| class RefitActorConfig(BaseModel, extra="allow"): |
There was a problem hiding this comment.
[DOC] docs/guides/single-controller.md should pick this knob up. Its "Config → behavior map" section carries a "Field definitions" list documenting async_rl.* knobs one at a time, including num_advantage_workers — the closest analog to this feature ("CPU Ray actors that run the advantage stage. 0 (the default) keeps it in the controller process"). refit_actor.enabled belongs in that list alongside it; right now it is documented only here and in the two exemplar YAMLs.
| algo_cfg = algo_config(master_config) | ||
|
|
||
| if ( | ||
| async_config.refit_actor.enabled |
There was a problem hiding this comment.
[TEST] This rejection has no test. tests/unit/weight_sync/test_refit_actor.py covers the config default (test_config_default_is_off) and the non-NCCL setup rejection (test_setup_rejects_non_nccl_transport), but nothing exercises the colocated guard, so it is the one new error path with no coverage. A pytest.raises(ValueError, match="non-colocated") over a MasterConfig with refit_actor.enabled=true and policy.generation.colocated.enabled=true would close it.
| (CollectiveWeightSynchronizer, NcclReshardWeightSynchronizer), | ||
| ): | ||
| raise ValueError( | ||
| "async_rl.refit_actor.enabled supports the NCCL refit transports " |
There was a problem hiding this comment.
[GUIDELINE] This precondition fires much later than it needs to. _move_synchronizer_to_refit_actor is called at setup.py:2467, after every worker group is up, after init_communicator, and after the setup-time sync_weights() — so an unsupported transport costs a full cluster bring-up before it reports.
The transport is a pure function of config: create_weight_synchronizer dispatches on policy.generation.backend, colocated, and generation.cfg["refit_transport"], all of which validate_single_controller_config can already see. The colocated half of this same precondition already lives there. Hoisting the backend/transport half next to it turns a multi-minute failure into a seconds-long one and keeps both halves of the feature's precondition in one place, which is what the error-handling guideline asks for ("prefer failing at setup() time over a deep-in-the-loop crash").
| return True | ||
|
|
||
|
|
||
| def _disarm_copies(root: Any, *, max_depth: int = 4) -> int: |
There was a problem hiding this comment.
[GUIDELINE] This is a reflective object-graph walk where an explicit list would do: it reads __dict__, __mro__ and vars(), installs instance attributes on whatever it encounters, and bounds itself with a magic max_depth=4 that truncates silently.
The objects that need disarming are known and named in the docstring, and in both NCCL transports they all sit at depth 1 off the synchronizer — _generation, _policy, _train_cluster, _inference_cluster. Disarming those explicitly would satisfy the "Avoid Reflection" rule in the linting-and-formatting guideline, drop the depth limit entirely, and fail loudly if one of the attributes is ever renamed, rather than quietly skipping it.
If you prefer to keep the generic walk, please at least log when the traversal hits max_depth with objects still unvisited. A copy sitting one level deeper than the limit shutting down the real vLLM workers is exactly the failure this function exists to prevent, and today it would happen with no signal.
What does this PR do ?
Adds
async_rl.refit_actor.enabled(off by default). When enabled, the SingleController's weight synchronizer runs in its own CPU Ray actor (RefitActor), so a busy controller event loop no longer delays the refit transport, and a slow refit no longer ties up a controller thread.Issues
Motivation
The SingleController runs the whole refit fan-out from its own process: the calls to every train worker and every generation leader, and the wait on all of their replies. The same process dispatches rollouts, finalizes groups and computes advantages. On large non-colocated runs we saw refits stretch to tens of seconds, and sometimes hang, while the controller was busy with rollout bookkeeping. This PR moves only the transport out of the controller process. Everything that decides when and with whom to refit stays where it is.
Design
WeightSynchronizer.sync_weights(the NCCL transfer)init_communicator/reconcile_communicator(communicator and plan rebuilds)RefitAbortedrecoverynemo_rl/weight_sync/refit_actor.py(new)RefitActor:@ray.remote(num_cpus=1, max_concurrency=4), placed on the controller's node (soft affinity). It holds the synchronizer.max_concurrency > 1letsstatus/reconcile_communicatoranswer while async_weightscall that outlived the deadline is still stuck.RemoteWeightSynchronizer: theWeightSynchronizerthe controller holds instead. It forwards each call and keeps two pieces of state in step between the controller's generation object and the actor's copy:restart_shard, the generation object is shipped again before the next call.is_stale/reconcile_communicator. The recovery path already forces a communicator rebuild after a failed refit.attach(generation, synchronizer_factory=...)to bind it to its own generation and to a factory that can rebuild a synchronizer if the actor has to be replaced._disarm_copies:VllmGeneration,PolicyandRayVirtualClustershut their real workers / placement groups down from__del__. Inside the RefitActor these objects are copies, so every object reachable from the synchronizer or a re-shipped generation whose class defines both__del__andshutdowngets a no-opshutdown. Without this, dropping the actor's old generation copy stopped the vLLM workers andray.killed the RefitActor itself. Ray handles and torch/numpy objects are not descended into.single_controller_utils/config.py:RefitActorConfig(BaseModel,enabled: bool = False) underAsyncRLConfig.refit_actor. Rejected with colocated generation (a colocated refit is the engine's own wake-up).single_controller_utils/setup.py:_move_synchronizer_to_refit_actorruns afterinit_communicatorand the setup-time refit, so setup itself is unchanged. NCCL transports only (collective,nccl_reshard). Anything else is a setup-timeValueError.single_controller.py: with a RefitActor,_sync_weights_withinawaits the actor'sObjectRefon the event loop under the same deadline (_sync_weights_in_refit_actor) instead of parking a thread. A timeout still raisesRefitAbortedinto the existing recovery.grpo_math_1B_megatron_single_controller.yamlandppo_math_1B_megatron_single_controller.yamldocument the key.refit_actor.pyis added topyrefly.toml.Usage
or on the command line:
++async_rl.refit_actor.enabled=true. With the flag off (the default), behaviour is unchanged.Results
GRPO on a MoE model with NeMo-Gym environments (32 prompts × 16 generations per step), non-colocated: 16 trainer GPUs and 12 vLLM GPUs,
nccl_reshardtransport, 6 steps. Same container, same config, only the flag differs. One run each.Refit time (controller's
_sync_weights, pause → transfer → resume), seconds:refit_actor.enabled=trueBefore your PR is "Ready for review"
Pre checks:
tests/unit/weight_sync/test_refit_actor.py(local CPU Ray, fake synchronizer/generation): forwarding, restarted-handle shipping, membership mirroring, dead-actor replacement, pickling +attach, factory replacement,_disarm_copies, re-shipping a self-shutting generation, quiet double shutdown, config default, non-NCCL rejection.tests/unit/weight_sync/test_refit_actor.py(14 passed) andtests/unit/single_controller/test_setup.py+test_resiliency_config.pypass in the NeMo RL container; ruff and pyrefly are clean. Functional: the two end-to-end runs above.Additional Information
sync_params_before_refit, pause, transfer, version stamp, resume) into the actor behind a second flag, returning per-phase timings. That would remove the remaining controller round trips between refit phases.