Skip to content

feat(sc): add elastic scheduler design, scheduling policy and replay tool - #5107

Draft
LoganVegnaSHOP wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
LoganVegnaSHOP:lv/elastic-scheduler
Draft

LoganVegnaSHOP wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
LoganVegnaSHOP:lv/elastic-scheduler

Conversation

@LoganVegnaSHOP

@LoganVegnaSHOP LoganVegnaSHOP commented Nov 6, 2026 •

Copy link
Copy Markdown

What does this PR do ?

Adds a design doc, a pure scheduling policy and an offline replay tool for elastic train/rollout cells on the SingleController. The training path is unchanged: nothing in the SC calls the policy yet, and there is no config.

Motivation

In agentic RL, rollout durations have a long tail. With a fixed split (async), rollout GPUs sit idle while the tail drains, and training GPUs wait for a batch. Colocate makes every GPU wait for the slowest rollout. QwenGyre (arXiv 2609.33848, §4) groups GPUs into cells. Each cell moves to training as soon as the remaining rollout work fits without it, and later cells join the batch that is already training.

Reported results:

  • up to 1.78x over Async and 1.85x over Colocate;
  • on the 12-step ablation, 16.12 h for elastic versus 19.23 h for the strongest colocated variant and 22.18 h for async with streaming;
  • the one-step-per-publication variant, which fits the SC's per-step refit, still beats both at 17.87 h.

What it does

  • docs/design-docs/elastic-scheduler.md: the full design and an 8-phase plan. It covers rerouting, offload/onload, the satellite weight pull, joining a running step, gradient reduction with changing membership (with an NCCL memory estimate and alternatives), staleness, checkpointing, failures, and where it doesn't help.
  • nemo_rl/algorithms/single_controller_utils/elastic_scheduler.py: a pure policy in the style of fleet_health.py. It implements the paper's eqs. 3–6: the capacity-gated launch rule, the waterlevel plus held-supply rule for moving cells one at a time with the core first, and core retention at burst end. The SC calls it in prompt groups; ElasticSchedulerPolicy.executions(groups) is the one place groups become executions, and launches never split a group.
  • tools/elastic_scheduler_replay.py: the policy's first consumer and the go/no-go measurement. It replays the rl.sc.generate_and_push span durations a real run recorded (OTLP/JSON or console exporter) through elastic cells, the best fixed split and colocate on the same cells. It reports step time, idle cell-time and switching overhead for each.

Usage

telemetry:
  enabled: true
  span_groups: per_step,per_prompt
uv run python tools/elastic_scheduler_replay.py --trace spans.jsonl \
    --num-cells 8 --cell-capacity 144 --group-size 16 --batch-groups 64 \
    --train-step-s 3600 --num-steps 12

Planned opt-in config (Phase 3, not in this PR): async_rl.elastic_scheduler.{enabled: false, num_cells, min_supply_fraction, supply_hold_s}.

Testing

  • Policy: 45 unit tests:
    • eq. 4/5/6 boundaries;
    • supply hold and its resets;
    • ordering and refusal of forced transitions;
    • group-unit conversion and never splitting a group;
    • a discrete-event simulation comparing elastic cells against switching the whole pool.
  • Replay tool: 18 unit tests:
    • both trace formats;
    • exact baseline step times;
    • the split sweep;
    • elastic beating colocate on a long tail, and not beating it on uniform durations;
    • the command line.
  • The full Sphinx HTML build (locked docs deps) has 0 warnings; the new page and toctree entry render.
  • Full CPU unit suite on macOS (default and --nemo-gym-only lanes), compared with clean main on the same machine: no new failures. The failures that remain also happen on main here (no triton/NCCL/GPU, missing Gym extras, Ray resource limits).

Additional Information

  • No controller, router or worker-group changes.
  • The replay is a filter, not a promise:
    • durations are replayed as recorded, regardless of which engine ran them or how loaded it was;
    • group-level slot accounting is conservative;
    • the fixed split's refit is not modelled as pausing generation, which favours the fixed split.
  • On a synthetic heavy-tail trace (K=4), elastic takes 0.77 h against colocate's 1.56 h, but only roughly matches a fixed split that runs one step ahead (0.72 h). Phase 2 must show a win on real traces before wiring starts.
  • NCCL memory for the K−1 prefix communicators is estimated from NCCL default buffer sizes, not measured: about 150 MiB per GPU per communicator, about 1 GiB on core GPUs at K=8. The doc recommends one parent communicator split with shared resources (splitShare=1) and notes that this would change how the existing communicator abort code behaves.
  • Where it doesn't help: env-bound rollouts (it saves GPU-hours, not wall-clock time), short or uniform rollouts, durations clustered at the timeout, small GPU budgets, training-bound workloads, and no global shuffle with multi-step bursts.

🤖 Generated with Claude Code

Add a design doc for elastic cells that switch between rollout and training
as the rollout tail drains (QwenGyre, section 4), and the first building
block: a pure scheduling policy implementing the launch rule, the
waterlevel and supply-hold rule for moving the next cell to training, and
core retention at burst end. The policy has no Ray, GPU or asyncio
dependency and nothing in the SingleController calls it yet, so there is
no behavior change.

Signed-off-by: Logan Vegna <logan.vegna@shopify.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 6, 2026
…ndary

The policy counts executions, as the paper does, while the SingleController
admits, launches and commits whole prompt groups. Give the policy the group
size and steps per burst instead of a precomputed burst dispatch, make
ElasticSchedulerPolicy.executions the single groups-to-executions
conversion, and add group-unit counters and launchable_groups, which never
splits a group across a launch decision.

Signed-off-by: Logan Vegna <logan.vegna@shopify.com>
Add tools/elastic_scheduler_replay.py, the offline go/no-go measurement for
elastic cells. It reads the per-group rl.sc.generate_and_push spans a
SingleController run records (OTLP/JSON from a collector file exporter, or
the console exporter's output), replays the durations through
ElasticScheduler, a fixed rollout/training split (every split tried unless
one is given) and colocate on the same cells, and reports projected step
time, idle cell-seconds and switching overhead for each.

Signed-off-by: Logan Vegna <logan.vegna@shopify.com>
…r risks

Document the offline replay as the go/no-go measurement with its trace
format, command and assumptions, plus a synthetic example. Explain the
group-to-execution boundary. Replace the open communicator-memory risk with
an estimate from NCCL default buffer sizes and five concrete alternatives,
recommending shared-resource splits. Spell out what the router and the
colocated SingleController checks already allow for MInf cells and what
each phase must change.

Signed-off-by: Logan Vegna <logan.vegna@shopify.com>
@LoganVegnaSHOP LoganVegnaSHOP changed the title feat(sc): add elastic scheduler design and scheduling policy feat(sc): add elastic scheduler design, scheduling policy and replay tool Nov 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant