Skip to content

fix(sc): Fix router replay with token capture for multi-turn rollouts - #4610

Merged
yuki-97 merged 7 commits into
NVIDIA-NeMo:mainfrom
mk-0:mkitov/sc-r3-fix
Oct 13, 2026
Merged

yuki-97 merged 7 commits into
NVIDIA-NeMo:mainfrom
mk-0:mkitov/sc-r3-fix

Conversation

@mk-0

@mk-0 mk-0 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Fix router replay with token capture for multi-turn rollouts. Before this fix, the recorded routed experts were wrong for the last token of each non-final assistant message.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

Affected configurations. The bug occurs only when all of these are true:

  • token_capture.enabled (only supported in single controller + vLLM).
  • policy.router_replay.enabled.
  • Multi-turn rollouts (more than one vLLM call per trajectory).

Token echo rollouts and single-turn rollouts are not affected.

Problem.

  1. vLLM records the experts of a token only when the token is an input. For a sequence of length S, vLLM returns a tensor with S − 1 rows. The last sampled token has no row.
  2. NeMo-RL makes a buffer with S rows (here). It fills each row with a default value, arange(k). Then it copies the S − 1 real rows. The last row keeps the default value.
  3. The next vLLM call uses the previous tokens as its prompt. Thus it calculates the real route for the previous last token, at row prev_len − 1.
  4. Token capture keeps only the new rows, experts[prev_len:]. It drops row prev_len − 1. Thus the default value from step 2 stays in the training data. (here)
  5. Token echo replaces this row (here). Token capture does not.

Effect.

  • The trainer uses experts 0 … k−1 at the last token of each non-final assistant message. This also affects the following tokens that attend to it.
  • The default value is a valid route, not -1. Thus the missing-route metrics do not show the problem.

Fix. For each vLLM call after the first, the fix keeps the route at row prev_len − 1. It writes this route into the last row of the previous message. The last token of the last message keeps the default value. This has no effect on training, because that token has no target and no token attends to it.

Verification. I reproduced the bug and tested the fix with a small Qwen MoE model. See this branch.

…pture

vLLM never routes a call's last sampled token; its route row is a
placeholder arange(top_k). The next call computes the real route during
prefill, at position prev_len - 1. With token capture, the vLLM worker
staged only rows prev_len onward, so in multi-turn rollouts the last
generated token of every non-final turn trained with the placeholder
route. The token echo path already patched this row.

- vllm_worker_async: a child call also stages row prev_len - 1.
- tq_token_sink / rollout_reassembler: accept the one extra row.
- route_assembly: execute_route_plan writes that row into the last
  position of the previous span. boundary_route_rows(prev_len) is the
  single rule used by all of these sites.

Direct and deferred route assembly share the executor, so both are fixed.
No Gym or route-plan schema change.

Verified at a9c3178 (the current nightly image): 50 unit tests pass;
an end-to-end GRPO step with token capture went from 0/8 to 8/8
turn-boundary tokens carrying vLLM's real route (repro:
mk-0/RL@mkitov/sc-r3-repro).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marat Kitov <mkitov@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@euronymous-aithal
euronymous-aithal requested review from pthombre and yuki-97 and removed request for pthombre October 9, 2026 14:43
@pthombre
pthombre requested a review from pjin-nvidia October 9, 2026 18:02
@pjin-nvidia

pjin-nvidia commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

thanks @mk-0 , lgtm!

one thing that also might be a good idea to add in this PR is to set a valid filler/fallback row value for routed experts, i.e. all -1's, instead of the current arange(topk):

https://github.com/mk-0/RL/blob/f43714698de4fadb7a8f1a5c69392d3b6971423f/nemo_rl/models/generation/vllm/utils.py#L321-L325

    default_route = torch.arange(
        routed.shape[2],
        dtype=routed_experts_dtype,
        device=device,
    )

vLLM never runs a sequence's last sampled token through the model, so it
has no route for it. pad_and_align_routed_expert_indices filled that row
with arange(top_k), a valid-looking route that forced experts 0..k-1 in
the Megatron forward and was invisible to the missing-route metrics.

Use the all--1 missing-route sentinel instead, so Megatron routes that
token itself. Batch padding past the sequence keeps its valid route, and
the vLLM missing-route stats and warning are unchanged (they already
excluded the final token). The finalizer's sentinel-token metric skips
each row's final token so it does not gain a constant baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marat Kitov <mkitov@nvidia.com>
@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 9, 2026
mk-0 and others added 2 commits October 9, 2026 23:17
…ctly

Initialize the routed-experts buffer with the missing-route sentinel
instead of arange(top_k), then copy in vLLM's real routes. Every row
without a recorded route (missing routes, the final token, batch padding)
now falls back to Megatron's router. Real-token behavior is unchanged;
batch padding never reaches router replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marat Kitov <mkitov@nvidia.com>
vLLM's route rows for a call's last sampled token are now the -1
sentinel, so the route-assembly test fixtures use it instead of the old
arange placeholder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marat Kitov <mkitov@nvidia.com>
@mk-0

mk-0 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @pjin-nvidia , I updated the snippet and the relevant tests.

@pthombre pthombre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and the clear write-up of the bug. The fix is correct for the shapes the new vLLM worker produces, and carrying the boundary row inside the already-digested routed_experts payload is a good approach.

Major issues flagged inline:

  • Executor infers the boundary row instead of reading it from the plan (route_assembly.py:158). Fragments without an extra leading row are classified one shape too low: a tail-mode fragment becomes sentinel and is dropped, which fails the existing test test_worker_coalesces_keys_and_replays_full_tail_and_placeholder, and a pre-fix plan resumed from a deferred-mode checkpoint writes the wrong token's route onto the parent's last token with no error.
  • Route plan schema version not bumped (route_plan.py:22). staged_route_len now means something different for child calls, so v2 plans should be rejected at decode rather than silently reinterpreted.

The remaining comments are test coverage for the new staging contract and the sentinel metric, a stale placeholder constant in the new test, and a small metric inconsistency.

Comment thread nemo_rl/experience/route_assembly.py Resolved
mk-0 and others added 2 commits October 12, 2026 22:27
- Rows without routes no longer count their final token as a sentinel,
  matching rows with routes.
- Route fixtures stage the -1 sentinel for each call's last token, so
  the metric's final-token exclusion is exercised.
- Add a case where a child call stages only its delta rows (no boundary
  row) and the finalizer rejects it with routed_len_mismatch.
- Fix a stale docstring in the route assembly test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marat Kitov <mkitov@nvidia.com>
…ng it

execute_route_plan assumed every non-root fragment leads with a boundary
row, inferred from the span position. A fragment without one (Gym's tail
shape, or a plan built before the boundary-row change) was misassembled
silently.

The finalizer now records it per span as RouteSpan.boundary_rows, set
from boundary_route_rows(prev_len) when the call staged routes, and the
executor reads it. The plan validates the field (0 or 1, never on the
first span or a span without staged routes) and bumps
ROUTE_PLAN_SCHEMA_VERSION to 3; decoding rejects older plans up front
with an explicit version error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marat Kitov <mkitov@nvidia.com>
@mk-0

mk-0 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @pthombre for the detailed review. I fixed all mentioned problems.

@pthombre

Copy link
Copy Markdown
Contributor

/ok to test 4782b6c

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 12, 2026 21:49
@pthombre

Copy link
Copy Markdown
Contributor

/ok to test 8df2f08

@pthombre pthombre added the CI:L1 Run doctests, unit tests, and functional tests label Oct 12, 2026
@terrykong terrykong added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Oct 12, 2026
@yuki-97
yuki-97 enabled auto-merge (squash) October 13, 2026 02:36
@yuki-97
yuki-97 merged commit fe1d97c into NVIDIA-NeMo:main Oct 13, 2026
142 of 144 checks passed
zyzhou5 added a commit to zyzhou5/RL that referenced this pull request Oct 13, 2026
…main

Merge main through 081b072 and retain the boundary row introduced by NVIDIA-NeMo#4610 in the native tensor capture path. Update capture and route-assembly tests for the boundary row and missing-route sentinel. Keep Gym and dependency pins unchanged from main.

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 19, 2026
…dary routes

NeMo RL now stages a multi-turn call's routes with one leading row for its
parent's last token (#4610): that token only gets a route once the next
call prefills it, and the reassembler writes that row back over the
parent's placeholder. The sink requires the extra row whenever prev_len > 0.

Generation-cut chunks keep their own convention, one route per staged
token, because only the resume path reads them back. The sink rejected
every such chunk of a multi-turn router-replay call, so the call could not
be cut. Chunks are now checked against their token delta alone.

The resumed call's canonical row is an ordinary capture row and carries the
boundary row its own prefill computed, as an uncut call would. The merge
that restores the prefix's original routes now starts after that row; it
used to expect one row fewer and failed the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 19, 2026
…dary routes

NeMo RL now stages a multi-turn call's routes with one leading row for its
parent's last token (#4610): that token only gets a route once the next
call prefills it, and the reassembler writes that row back over the
parent's placeholder. The sink requires the extra row whenever prev_len > 0.

Generation-cut chunks keep their own convention, one route per staged
token, because only the resume path reads them back. The sink rejected
every such chunk of a multi-turn router-replay call, so the call could not
be cut. Chunks are now checked against their token delta alone.

The resumed call's canonical row is an ordinary capture row and carries the
boundary row its own prefill computed, as an uncut call would. The merge
that restores the prefix's original routes now starts after that row; it
used to expect one row fewer and failed the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 21, 2026
…dary routes

NeMo RL now stages a multi-turn call's routes with one leading row for its
parent's last token (#4610): that token only gets a route once the next
call prefills it, and the reassembler writes that row back over the
parent's placeholder. The sink requires the extra row whenever prev_len > 0.

Generation-cut chunks keep their own convention, one route per staged
token, because only the resume path reads them back. The sink rejected
every such chunk of a multi-turn router-replay call, so the call could not
be cut. Chunks are now checked against their token delta alone.

The resumed call's canonical row is an ordinary capture row and carries the
boundary row its own prefill computed, as an uncut call would. The merge
that restores the prefix's original routes now starts after that row; it
used to expect one row fewer and failed the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 22, 2026
…dary routes

NeMo RL now stages a multi-turn call's routes with one leading row for its
parent's last token (#4610): that token only gets a route once the next
call prefills it, and the reassembler writes that row back over the
parent's placeholder. The sink requires the extra row whenever prev_len > 0.

Generation-cut chunks keep their own convention, one route per staged
token, because only the resume path reads them back. The sink rejected
every such chunk of a multi-turn router-replay call, so the call could not
be cut. Chunks are now checked against their token delta alone.

The resumed call's canonical row is an ordinary capture row and carries the
boundary row its own prefill computed, as an uncut call would. The merge
that restores the prefix's original routes now starts after that row; it
used to expect one row fewer and failed the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants