Repository navigation
fix(sc): Fix router replay with token capture for multi-turn rollouts - #4610
Conversation
…pture vLLM never routes a call's last sampled token; its route row is a placeholder arange(top_k). The next call computes the real route during prefill, at position prev_len - 1. With token capture, the vLLM worker staged only rows prev_len onward, so in multi-turn rollouts the last generated token of every non-final turn trained with the placeholder route. The token echo path already patched this row. - vllm_worker_async: a child call also stages row prev_len - 1. - tq_token_sink / rollout_reassembler: accept the one extra row. - route_assembly: execute_route_plan writes that row into the last position of the previous span. boundary_route_rows(prev_len) is the single rule used by all of these sites. Direct and deferred route assembly share the executor, so both are fixed. No Gym or route-plan schema change. Verified at a9c3178 (the current nightly image): 50 unit tests pass; an end-to-end GRPO step with token capture went from 0/8 to 8/8 turn-boundary tokens carrying vLLM's real route (repro: mk-0/RL@mkitov/sc-r3-repro). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
|
thanks @mk-0 , lgtm! one thing that also might be a good idea to add in this PR is to set a valid filler/fallback row value for routed experts, i.e. all -1's, instead of the current |
vLLM never runs a sequence's last sampled token through the model, so it has no route for it. pad_and_align_routed_expert_indices filled that row with arange(top_k), a valid-looking route that forced experts 0..k-1 in the Megatron forward and was invisible to the missing-route metrics. Use the all--1 missing-route sentinel instead, so Megatron routes that token itself. Batch padding past the sequence keeps its valid route, and the vLLM missing-route stats and warning are unchanged (they already excluded the final token). The finalizer's sentinel-token metric skips each row's final token so it does not gain a constant baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
…ctly Initialize the routed-experts buffer with the missing-route sentinel instead of arange(top_k), then copy in vLLM's real routes. Every row without a recorded route (missing routes, the final token, batch padding) now falls back to Megatron's router. Real-token behavior is unchanged; batch padding never reaches router replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
vLLM's route rows for a call's last sampled token are now the -1 sentinel, so the route-assembly test fixtures use it instead of the old arange placeholder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
|
Thanks @pjin-nvidia , I updated the snippet and the relevant tests. |
pthombre
left a comment
There was a problem hiding this comment.
Thanks for the PR and the clear write-up of the bug. The fix is correct for the shapes the new vLLM worker produces, and carrying the boundary row inside the already-digested routed_experts payload is a good approach.
Major issues flagged inline:
- Executor infers the boundary row instead of reading it from the plan (
route_assembly.py:158). Fragments without an extra leading row are classified one shape too low: a tail-mode fragment becomes sentinel and is dropped, which fails the existing testtest_worker_coalesces_keys_and_replays_full_tail_and_placeholder, and a pre-fix plan resumed from a deferred-mode checkpoint writes the wrong token's route onto the parent's last token with no error. - Route plan schema version not bumped (
route_plan.py:22).staged_route_lennow means something different for child calls, so v2 plans should be rejected at decode rather than silently reinterpreted.
The remaining comments are test coverage for the new staging contract and the sentinel metric, a stale placeholder constant in the new test, and a small metric inconsistency.
- Rows without routes no longer count their final token as a sentinel, matching rows with routes. - Route fixtures stage the -1 sentinel for each call's last token, so the metric's final-token exclusion is exercised. - Add a case where a child call stages only its delta rows (no boundary row) and the finalizer rejects it with routed_len_mismatch. - Fix a stale docstring in the route assembly test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
…ng it execute_route_plan assumed every non-root fragment leads with a boundary row, inferred from the span position. A fragment without one (Gym's tail shape, or a plan built before the boundary-row change) was misassembled silently. The finalizer now records it per span as RouteSpan.boundary_rows, set from boundary_route_rows(prev_len) when the call staged routes, and the executor reads it. The plan validates the field (0 or 1, never on the first span or a span without staged routes) and bumps ROUTE_PLAN_SCHEMA_VERSION to 3; decoding rejects older plans up front with an explicit version error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
|
Thanks @pthombre for the detailed review. I fixed all mentioned problems. |
|
/ok to test 4782b6c |
|
/ok to test 8df2f08 |
…main Merge main through 081b072 and retain the boundary row introduced by NVIDIA-NeMo#4610 in the native tensor capture path. Update capture and route-assembly tests for the boundary row and missing-route sentinel. Keep Gym and dependency pins unchanged from main. Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>
…dary routes NeMo RL now stages a multi-turn call's routes with one leading row for its parent's last token (#4610): that token only gets a route once the next call prefills it, and the reassembler writes that row back over the parent's placeholder. The sink requires the extra row whenever prev_len > 0. Generation-cut chunks keep their own convention, one route per staged token, because only the resume path reads them back. The sink rejected every such chunk of a multi-turn router-replay call, so the call could not be cut. Chunks are now checked against their token delta alone. The resumed call's canonical row is an ordinary capture row and carries the boundary row its own prefill computed, as an uncut call would. The merge that restores the prefix's original routes now starts after that row; it used to expect one row fewer and failed the call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…dary routes NeMo RL now stages a multi-turn call's routes with one leading row for its parent's last token (#4610): that token only gets a route once the next call prefills it, and the reassembler writes that row back over the parent's placeholder. The sink requires the extra row whenever prev_len > 0. Generation-cut chunks keep their own convention, one route per staged token, because only the resume path reads them back. The sink rejected every such chunk of a multi-turn router-replay call, so the call could not be cut. Chunks are now checked against their token delta alone. The resumed call's canonical row is an ordinary capture row and carries the boundary row its own prefill computed, as an uncut call would. The merge that restores the prefix's original routes now starts after that row; it used to expect one row fewer and failed the call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…dary routes NeMo RL now stages a multi-turn call's routes with one leading row for its parent's last token (#4610): that token only gets a route once the next call prefills it, and the reassembler writes that row back over the parent's placeholder. The sink requires the extra row whenever prev_len > 0. Generation-cut chunks keep their own convention, one route per staged token, because only the resume path reads them back. The sink rejected every such chunk of a multi-turn router-replay call, so the call could not be cut. Chunks are now checked against their token delta alone. The resumed call's canonical row is an ordinary capture row and carries the boundary row its own prefill computed, as an uncut call would. The merge that restores the prefix's original routes now starts after that row; it used to expect one row fewer and failed the call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…dary routes NeMo RL now stages a multi-turn call's routes with one leading row for its parent's last token (#4610): that token only gets a route once the next call prefills it, and the reassembler writes that row back over the parent's placeholder. The sink requires the extra row whenever prev_len > 0. Generation-cut chunks keep their own convention, one route per staged token, because only the resume path reads them back. The sink rejected every such chunk of a multi-turn router-replay call, so the call could not be cut. Chunks are now checked against their token delta alone. The resumed call's canonical row is an ordinary capture row and carries the boundary row its own prefill computed, as an uncut call would. The merge that restores the prefix's original routes now starts after that row; it used to expect one row fewer and failed the call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
What does this PR do ?
Fix router replay with token capture for multi-turn rollouts. Before this fix, the recorded routed experts were wrong for the last token of each non-final assistant message.
Before your PR is "Ready for review"
Pre checks:
Additional Information
Affected configurations. The bug occurs only when all of these are true:
token_capture.enabled(only supported in single controller + vLLM).policy.router_replay.enabled.Token echo rollouts and single-turn rollouts are not affected.
Problem.
S, vLLM returns a tensor withS − 1rows. The last sampled token has no row.Srows (here). It fills each row with a default value,arange(k). Then it copies theS − 1real rows. The last row keeps the default value.prev_len − 1.experts[prev_len:]. It drops rowprev_len − 1. Thus the default value from step 2 stays in the training data. (here)Effect.
0 … k−1at the last token of each non-final assistant message. This also affects the following tokens that attend to it.-1. Thus the missing-route metrics do not show the problem.Fix. For each vLLM call after the first, the fix keeps the route at row
prev_len − 1. It writes this route into the last row of the previous message. The last token of the last message keeps the default value. This has no effect on training, because that token has no target and no token attends to it.Verification. I reproduced the bug and tested the fix with a small Qwen MoE model. See this branch.