Skip to content

feat(single-controller): optionally run weight synchronizer in separate Actor - #5182

Open
afarjallah-nv wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
afarjallah-nv:feat/single-controller-refit-actor
Open

afarjallah-nv wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
afarjallah-nv:feat/single-controller-refit-actor

Conversation

@afarjallah-nv

Copy link
Copy Markdown

What does this PR do ?

Adds async_rl.refit_actor.enabled (off by default). When enabled, the SingleController's weight synchronizer runs in its own CPU Ray actor (RefitActor), so a busy controller event loop no longer delays the refit transport, and a slow refit no longer ties up a controller thread.

Issues

Motivation

The SingleController runs the whole refit fan-out from its own process: the calls to every train worker and every generation leader, and the wait on all of their replies. The same process dispatches rollouts, finalizes groups and computes advantages. On large non-colocated runs we saw refits stretch to tens of seconds, and sometimes hang, while the controller was busy with rollout bookkeeping. This PR moves only the transport out of the controller process. Everything that decides when and with whom to refit stays where it is.

Design

Stays in the SingleController Moves to the RefitActor
Rollout gate, generation pause/resume (#3929), KV-cache invalidation, weight-version stamping WeightSynchronizer.sync_weights (the NCCL transfer)
Refit membership decisions, fleet health init_communicator / reconcile_communicator (communicator and plan rebuilds)
Refit deadline and RefitAborted recovery
  • nemo_rl/weight_sync/refit_actor.py (new)
    • RefitActor: @ray.remote(num_cpus=1, max_concurrency=4), placed on the controller's node (soft affinity). It holds the synchronizer. max_concurrency > 1 lets status / reconcile_communicator answer while a sync_weights call that outlived the deadline is still stuck.
    • RemoteWeightSynchronizer: the WeightSynchronizer the controller holds instead. It forwards each call and keeps two pieces of state in step between the controller's generation object and the actor's copy:
      • Worker handles. After restart_shard, the generation object is shipped again before the next call.
      • Refit membership. The membership the actor builds is applied to the controller's generation, which pause/resume/KV invalidation dispatch through.
    • A dead RefitActor is replaced on the next is_stale / reconcile_communicator. The recovery path already forces a communicator rebuild after a failed refit.
    • Pickles as its actor handle only. The SingleControllerActor receives the proxy inside its constructor args and calls attach(generation, synchronizer_factory=...) to bind it to its own generation and to a factory that can rebuild a synchronizer if the actor has to be replaced.
    • _disarm_copies: VllmGeneration, Policy and RayVirtualCluster shut their real workers / placement groups down from __del__. Inside the RefitActor these objects are copies, so every object reachable from the synchronizer or a re-shipped generation whose class defines both __del__ and shutdown gets a no-op shutdown. Without this, dropping the actor's old generation copy stopped the vLLM workers and ray.killed the RefitActor itself. Ray handles and torch/numpy objects are not descended into.
  • single_controller_utils/config.py: RefitActorConfig (BaseModel, enabled: bool = False) under AsyncRLConfig.refit_actor. Rejected with colocated generation (a colocated refit is the engine's own wake-up).
  • single_controller_utils/setup.py: _move_synchronizer_to_refit_actor runs after init_communicator and the setup-time refit, so setup itself is unchanged. NCCL transports only (collective, nccl_reshard). Anything else is a setup-time ValueError.
  • single_controller.py: with a RefitActor, _sync_weights_within awaits the actor's ObjectRef on the event loop under the same deadline (_sync_weights_in_refit_actor) instead of parking a thread. A timeout still raises RefitAborted into the existing recovery.
  • Exemplar configs grpo_math_1B_megatron_single_controller.yaml and ppo_math_1B_megatron_single_controller.yaml document the key. refit_actor.py is added to pyrefly.toml.

Usage

async_rl:
  refit_actor:
    enabled: true   # requires policy.generation.colocated.enabled=false and an NCCL refit transport

or on the command line: ++async_rl.refit_actor.enabled=true. With the flag off (the default), behaviour is unchanged.

Results

GRPO on a MoE model with NeMo-Gym environments (32 prompts × 16 generations per step), non-colocated: 16 trainer GPUs and 12 vLLM GPUs, nccl_reshard transport, 6 steps. Same container, same config, only the flag differs. One run each.

Refit time (controller's _sync_weights, pause → transfer → resume), seconds:

Refit 0 (initial) 1 2 3 4 5 6 Sum
Flag off 3.615 2.193 2.315 1.908 2.382 4.585 1.550 18.548
refit_actor.enabled=true 2.941 1.306 1.752 1.869 1.368 2.491 1.409 13.136
Speedup 1.23× 1.68× 1.32× 1.02× 1.74× 1.84× 1.10× 1.41×
  • Every refit was faster with the actor. The train-side broadcast inside the transfer is identical in both runs (~0.2 s), so the saving is in the controller-side fan-out and wait.
  • Both runs completed all 6 steps. Training, logprob and checkpoint times are unchanged (policy_training 475 s vs 474 s over 6 steps). Generated and trained tokens per second of step time match within run-to-run noise.
  • At this scale refit is under 1% of step time, so end-to-end step time is dominated by generation length. The flag is aimed at large runs, where refit fan-out from a loaded controller is a larger share of the step.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests? tests/unit/weight_sync/test_refit_actor.py (local CPU Ray, fake synchronizer/generation): forwarding, restarted-handle shipping, membership mirroring, dead-actor replacement, pickling + attach, factory replacement, _disarm_copies, re-shipping a self-shutting generation, quiet double shutdown, config default, non-NCCL rejection.
  • Did you run the unit tests and functional tests locally? Unit tests: tests/unit/weight_sync/test_refit_actor.py (14 passed) and tests/unit/single_controller/test_setup.py + test_resiliency_config.py pass in the NeMo RL container; ruff and pyrefly are clean. Functional: the two end-to-end runs above.
  • Did you add or update any necessary documentation? Config docstring plus the exemplar YAMLs.

Additional Information

  • Possible follow-up: move the whole refit transaction (sync_params_before_refit, pause, transfer, version stamp, resume) into the actor behind a second flag, returning per-phase timings. That would remove the remaining controller round trips between refit phases.
  • Not covered by this PR: colocated generation and non-NCCL transports (IPC, checkpoint-engine, sglang, megatron inference). They are rejected at config/setup time.

…separate RefitActor

Add async_rl.refit_actor.enabled (off by default). When enabled, the
SingleController's weight synchronizer runs in its own CPU Ray actor on
the controller's node, so a busy controller event loop no longer delays
the refit transport and a slow refit no longer occupies a controller
thread.

Only the transport moves (sync_weights, init/reconcile_communicator).
The rollout gate, generation pause/resume, refit membership decisions,
the refit deadline and recovery stay in the controller.

- RefitActor holds the synchronizer; RemoteWeightSynchronizer is the
  controller-side proxy. It re-ships the generation object when worker
  handles change, mirrors the refit membership the actor builds, and
  replaces a dead actor on the next is_stale / reconcile_communicator.
- The proxy pickles as its actor handle only; the SingleControllerActor
  attaches it to its own generation plus a synchronizer factory.
- Copies of the generation, policy and clusters inside the actor get a
  no-op shutdown so their __del__ safety nets cannot stop the real
  workers or kill the actor.
- Rejected with colocated generation; NCCL transports only.
- The controller awaits the actor's ObjectRef under the existing refit
  deadline; a timeout still raises RefitAborted into recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: afarjallah <afarjallah@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@afarjallah-nv afarjallah-nv changed the title feat(single-controller): optionally run the weight synchronizer in a separate RefitActor feat(single-controller): optionally run the weight synchronizer in a separate Actor Nov 12, 2026
@afarjallah-nv afarjallah-nv self-assigned this Nov 12, 2026
@afarjallah-nv afarjallah-nv changed the title feat(single-controller): optionally run the weight synchronizer in a separate Actor feat(single-controller): optionally run weight synchronizer in separate Actor Nov 12, 2026
@saumishr
saumishr requested a review from mehraakash November 12, 2026 19:57

@saumishr saumishr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the proxy's interface coverage, the dead-actor recovery path, and the replacement-synchronizer bootstrap against the source; all three are correct, so the findings below are mostly process and docs.

Two items that have no natural line to attach to:

[CI] This PR has no CI:* label. Every PR needs exactly one or the quality-check job stays red. CI:L0 fits ("new code"), or CI:L1 for the functional tier given this touches the refit path — the feature is off by default, so L0 is defensible.

[Question] Actor placement diverges from the nearest analog. RefitActor soft-pins to the controller's node to keep the RPC hop local, while AdvantageActor deliberately does the opposite and falls back to "SPREAD" with the comment "Ray's hybrid default packs a CPU-only actor onto the driver's node -- beside the controller it is relieving." Both rationales are reasonable for their workloads (I/O-bound fan-out vs. CPU-heavy tensor work), but since the sibling documents the opposite tradeoff, a sentence confirming the choice would help.

Also minor: is_stale now makes two blocking ray.get calls (ping + status) and is read synchronously inside the async _run_pumps at single_controller.py:883. It is startup-only so it costs nothing in steady state, but it is a blocking call on the event loop in a PR about not blocking the controller.


Things I expected to be problems and confirmed are fine, so they don't get re-litigated later: the proxy implements the complete WeightSynchronizer surface, so nothing falls through to the ABC; _generation and _built_membership do exist on both NCCL transports; dead-actor recovery works because RayActorError is already in the recovery except at line 4934 and reconcile_communicator replaces the actor before forwarding; a replacement synchronizer is never init_communicator'd, but reconcile_communicator(force=True) handles _built_membership is None by assuming the full fleet and rebuilding, and a fresh CollectiveWeightSynchronizer starts _stale = True; _build_refit_synchronizer matches both create_weight_synchronizer call sites argument for argument; and the max_concurrency=4 race between a stuck sync_weights and a recovery reconcile_communicator is the same race the in-process path already has with its orphaned thread. pyrefly.toml, the # pragma: no cover, copyright headers and sign-off are all in order, and the two edited exemplars have no counterpart in tests/unit/reference_configs/, so test_config_v2.py needs no update.

return self


class RefitActorConfig(BaseModel, extra="allow"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[DOC] docs/guides/single-controller.md should pick this knob up. Its "Config → behavior map" section carries a "Field definitions" list documenting async_rl.* knobs one at a time, including num_advantage_workers — the closest analog to this feature ("CPU Ray actors that run the advantage stage. 0 (the default) keeps it in the controller process"). refit_actor.enabled belongs in that list alongside it; right now it is documented only here and in the two exemplar YAMLs.

algo_cfg = algo_config(master_config)

if (
async_config.refit_actor.enabled

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[TEST] This rejection has no test. tests/unit/weight_sync/test_refit_actor.py covers the config default (test_config_default_is_off) and the non-NCCL setup rejection (test_setup_rejects_non_nccl_transport), but nothing exercises the colocated guard, so it is the one new error path with no coverage. A pytest.raises(ValueError, match="non-colocated") over a MasterConfig with refit_actor.enabled=true and policy.generation.colocated.enabled=true would close it.

(CollectiveWeightSynchronizer, NcclReshardWeightSynchronizer),
):
raise ValueError(
"async_rl.refit_actor.enabled supports the NCCL refit transports "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[GUIDELINE] This precondition fires much later than it needs to. _move_synchronizer_to_refit_actor is called at setup.py:2467, after every worker group is up, after init_communicator, and after the setup-time sync_weights() — so an unsupported transport costs a full cluster bring-up before it reports.

The transport is a pure function of config: create_weight_synchronizer dispatches on policy.generation.backend, colocated, and generation.cfg["refit_transport"], all of which validate_single_controller_config can already see. The colocated half of this same precondition already lives there. Hoisting the backend/transport half next to it turns a multi-minute failure into a seconds-long one and keeps both halves of the feature's precondition in one place, which is what the error-handling guideline asks for ("prefer failing at setup() time over a deep-in-the-loop crash").

return True


def _disarm_copies(root: Any, *, max_depth: int = 4) -> int:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[GUIDELINE] This is a reflective object-graph walk where an explicit list would do: it reads __dict__, __mro__ and vars(), installs instance attributes on whatever it encounters, and bounds itself with a magic max_depth=4 that truncates silently.

The objects that need disarming are known and named in the docstring, and in both NCCL transports they all sit at depth 1 off the synchronizer — _generation, _policy, _train_cluster, _inference_cluster. Disarming those explicitly would satisfy the "Avoid Reflection" rule in the linting-and-formatting guideline, drop the depth limit entirely, and fail loudly if one of the attributes is ever renamed, rather than quietly skipping it.

If you prefer to keep the generic walk, please at least log when the traversal hits max_depth with objects still unvisited. A copy sitting one level deeper than the limit shutting down the real vLLM workers is exactly the failure this function exists to prevent, and today it would happen with no signal.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants