Skip to content

feat(dynamo): serve NeMo-Gym through the Dynamo frontend - #4979

Open
jthomson04 wants to merge 125 commits into
NVIDIA-NeMo:mainfrom
jthomson04:jthomson04/dynamo-gym
Open

jthomson04 wants to merge 125 commits into
NVIDIA-NeMo:mainfrom
jthomson04:jthomson04/dynamo-gym

Conversation

@jthomson04

@jthomson04 jthomson04 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What

With policy.generation.dynamo: true, NeMo-Gym's policy model calls the Dynamo frontend instead of each vLLM shard's HTTP server. Gym's chat calls take the native server's steps, run by the native server's code, token capture included.

Based on main: #4905 merged as f488a0c6eb, and this branch has merged main since.

How a Gym call is served

Dynamo's frontend runs NeMo RL's own chat engine (Dynamo's chat_engine_factory hook) instead of Dynamo's Rust renderer and parsers. /v1/completions, which NeMo RL's own rollouts use, is unchanged.

  1. Frontend, render pool (dynamo/vllm/render_pool.py): spawned processes build vLLM's renderer and chat serving from the engine args and the policy.generation config each engine publishes when it registers, parser plugins and NeMo RL's video loader included. A worker renders the call and decides how a token-capture call's prefix is served. A worker that dies (an OOM kill, say) fails only the calls it held; the pool restarts.
  2. Frontend, chat engine (dynamo/vllm/chat_engine.py): checks on-policy sampling, then prepares the prompt (prepare_prompt): it reads a captured prefix from TQ, splices it in at the model's turn ends and clamps the output budget. The final token IDs go through the KV router, with the prepared prompt in extra_args. A measurement call (ng_measure) gets the prompt length back without reaching an engine.
  3. Engine (the Gym handler on DynamoVllmAdapter): start_capture (a restore, the media snapshot, begin_call), generation with the SamplingParams vLLM's chat serving builds, sampler top-k for score centering, routes, and one TQ write (save_capture). A call the engine refuses comes back with the status the native server answers it with.
  4. Frontend, render pool: builds the reply with vLLM's reasoning and tool parsers. The call's token fields, or a captured call's commit coordinates, go back in nvext.nemo_rl.

The steps after the render are one set of functions that both paths call (vllm/prompt_and_capture.py): prepare_prompt, start_capture, save_capture, and one error map (error_reply) for the status, the error body, the failed-capture reason and Gym's acknowledgement. The native server runs them in the worker; behind Dynamo the frontend prepares the prompt and the engine runs the rest. They come from Terry Kong's sketch for review comment C88 (jthomson04#13), which is the first of this round's commits.

frontend_args.render_workers (default 8) sets the pool size. One worker renders about 640 single-turn calls/s, 45 for a 21K-token agentic history (22 with its prefix splice), or 200 with 1–4 images (one Grace core on GB200).

When the engines expose chat (vllm_cfg.expose_http_server, which NeMo-Gym requires and the native worker starts its server on), the workers are built while the engines load (e1c510f8dc). The frontend starts them at launch, and each imports vLLM. An engine sends its settings as soon as it has its engine args, before it builds the engine, and the workers build from them. At registration the chat engine keeps the workers if the engines register the same policy.generation, and replaces them otherwise. On the local Gym capture test the workers are ready 10 to 12 s before the engine registers, the first chat call does not wait for them, and step 1 takes 8.0 s through Dynamo against 7.4 s native. The CMH runs under Validation predate this: there the workers started at registration, and the first step of a Gym config was 13 to 57 s slower than native. A run without the flag starts none.

Rollout recovery

rollout_recovery.target_level: turn and prefix (#4506) work through Dynamo.

With prefix, a checkpoint cuts each running call and a restore resumes it from the tokens the cut saved. The engines hold the calls, so both halves run there, with the native server's code:

  • The cut. Gym posts it beside the URL its calls go to, which is the frontend. A Dynamo frontend route can only be a synchronous GET, so the frontend process serves the native route on a second port (dynamo/vllm/generation_cut.py) and sends the cut to every engine over the request plane. Each engine answers with the calls it cut, and the answers are merged into the one receipt Gym expects. The frontend sends its deadline with the cut, so an engine stops cutting in time for its answer to count. [FEATURE]: Frontend routes with POST, a request body and async handlers ai-dynamo/dynamo#18486 asks for POST routes, which would remove the port.
  • The resume. The frontend renders and routes a re-sent call like any other. The engine then reads the saved rows, replaces the prompt with them and lowers the output budget. A cut it cannot use (rows gone, or newer than its weights) is dropped and the call regenerates, as natively. The render pool puts the saved text back before vLLM's parsers run and counts the saved tokens as output. A resumed call that carries media reprocesses it and checks it against the digests the cut saved (feat(checkpoint): recover multimodal generation prefixes #4583), before begin_call: changed pixels are refused with retained_media_changed, as natively.

The resolve is in the engine because it needs the engine's weight version and capture state. The cost is that the KV router scores a resumed call on the prompt without its saved tokens.

Changes

Shared with the native server (vLLM only; Megatron is a follow-up):

  • The chat server moves out of the async worker into vllm/chat_server.py, unchanged.
  • resolve_capture_input (the capture decision for a Gym candidate), on_policy_sampling, turn_end_token_ids and clamp_max_tokens are each one function both paths call. splice_prefix_tokens and replace_prefix_tokens now require the turn-end set.
  • The steps after the render (vllm/prompt_and_capture.py, above) and the reply fixes for a resumed call (ResumedCallReplyMixin) are shared. feat(checkpoint): recover active vLLM generation prefixes #4506's generation-cut route takes any backend with a bearer token and a run_cut.
  • The native route answers off-policy sampling with a 400 instead of a 500.
  • The native endpoint serves every call it serves on main. Only the Dynamo path refuses stream: true and n other than 1, with a 400 and before anything runs; NeMo-Gym sends neither. logprobs without top_logprobs (omitted or null) is served as 0 on both paths, as vLLM 0.33 serves it, and test_vllm_http_logprobs_contract is unchanged from main.
  • Statuses: a full queue answers 503 and any other vLLM client error its own 4xx, including one raised while the request body is parsed. Both were a 500.
  • reasoning_ended is read from the template's render, not the spliced prompt, so a multi-turn call with structured output and a reasoning parser can reason before its grammar applies.
  • A resumed call splices and clamps before its restore is resolved; the result is the same.
  • A native chat call whose prompt fills the context exactly, with no max_tokens, is refused with the "maximum context length" message, which Gym marks context_length_exceeded, instead of vLLM's "max_tokens must be at least 1". The same calls are refused as before.

Gym contract: GymRequestFields, GymCallExtras and GymCallResult (models/generation/gym_call.py) define the fields NeMo RL and Gym exchange; the native request classes inherit GymRequestFields. Gym's Dynamo dialect sends the extras in one nemo_rl:gym: annotation and reads the result from nvext.nemo_rl. Through Dynamo, a context-length refusal acknowledges the capture call at the end of the error message, and Gym moves it back to x-ng-capture-refused.

Token capture uses Gym's dynamo_worker backend. Forest training and score centering work through Dynamo. frontend_args.session_affinity_ttl_secs (default 300) keeps a capture session on one engine from Dynamo 1.7 (#5115); Dynamo 1.5.1 drops a chat call's session id before the router reads it.

Config:

  • dynamo_cfg.worker_args stays removed, as in feat(dynamo)!: serve NeMo RL's own vLLM rollouts through Dynamo's backend SDK #4905, and is rejected at startup: Gym chat uses the native vllm_cfg chat settings.
  • New frontend_args.render_workers.
  • extra_cli_args may not set --dyn-chat-processor or --custom-jinja-template.
  • DYN_IGNORE_OPENAI_FE_UNSUPPORTED_FIELDS=0 is pinned, so the frontend never drops an unknown field silently.

Also rejected at startup with dynamo: true:

  • async_rl.generation_router.enabled: the frontend already routes the fleet.
  • bad_words in extra_body or sampling_overrides on the policy model: the frontend answers it with a 400.
  • stop_token_ids, allowed_token_ids, bad_words_token_ids, cache_salt, logprob_token_ids or detokenize in either block on the policy model: the frontend accepts them but never hands them to a Python chat engine.
  • vllm_cfg.skip_tokenizer_init: true with NeMo-Gym: the engine detokenizes Gym's replies.
  • vllm_cfg.http_api_server_count above 1 (feat(vllm): share one engine across multiple HTTP frontends #5074's extra vLLM HTTP frontends per engine): Gym's calls go through the Dynamo frontend, so those processes would serve none of them. frontend_args.render_workers is the Dynamo-side setting.

Differences from native, also in the guide:

  • The frontend refuses more than 32 stop strings, and vLLM request fields its own request type lacks (structured_outputs, priority, ...), with a 400.
  • gpt-oss (Harmony) chat calls are refused with a 400: vLLM renders them without the step NeMo RL's capture and splice run in.
  • A context-length refusal of a capture call is acknowledged in the error message, not a header (see above).
  • A call's processed media travels to the engine inside the request.

CI:

  • L1_Functional_Tests_Dynamo.sh runs every overlay in fast mode: math, Gym with and without capture, reload refit, context compaction (forest training), router replay with deferred routes, turn recovery, and prefix recovery (C94).
  • The swe1 Dynamo + Gym nightly is re-enabled.

Merge order

  1. build(gym): bump the Gym submodule #5195, the Gym bump. Merged to main as 8f3595a112, and merged into this branch at e3aec3d1ce, with its code_gen fix.
  2. feat(vllm_model): Dynamo frontend dialect and dynamo_worker capture handler Gym#4817. Merged as 440104210.
  3. This PR, pinned to Gym main at 9e2e5d0ad: the Gym#4817 merge plus five later commits.
  4. feat(checkpoint): recover multimodal generation prefixes #4583 (multimodal prefix recovery). Merged to main as bb9ed784e4, and merged into this branch at 7e4fbd5ea4. This PR lands second, so start_capture now makes feat(checkpoint): recover multimodal generation prefixes #4583's restored-media call (C92), which both paths run.

Against the previous pin (6dc68d2fa, on Gym#4817's branch), the new one adds 39 other Gym main commits. The vllm_model server is the same at the merge commit, and Gym's dependencies are the same, so uv.lock does not change. The pin contains the commit #5195 pins (23557b7c2).

Since the functional and cluster runs under Validation, Gym#4817's review added four commits to the dialect:

  • reasoning_field: both is refused with dynamo_frontend only when the Gym server replays reasoning. NeMo RL's setup sets reasoning_content, so its policy model never reached the check.
  • A top-k capture call that also sets logprob_token_ids keeps Gym's HTTP 400 through Dynamo. This PR's startup check covers the same field only in NeMo RL's own policy-model overlay (extra_body and sampling_overrides).
  • forward_session_id_as_conversation_id is refused with dynamo_frontend, because the frontend rejects the conversation_params field it sends. NeMo RL reads that field only in its TensorRT-LLM server.
  • Gym's output-budget refit no longer asks the frontend for a /tokenize count. This PR's chat engine lowers the budget itself, so the count was never used.
  • Gym's vLLM guide documents the dialect, following this PR's guide, and Gym's tests cover every rejected setting, the empty annotation and the Dynamo capture handler.

None of these changes a call this PR's recipes make.

On the re-pin (766f81d004), run locally:

  • uv lock --check passes.
  • vLLM lane with Gym installed: 66 passed.
  • Gym lane: 1,191 passed and 14 errors. The 14 are the cases of test_l0_gym_environments_roll_out_through_nemo_rl. They fail at setup because of the bump, not this PR: since refactor(code-gen): replace legacy Ray execution with sandboxes Gym#4958, Gym's code_gen server requires sandbox_config.image. build(gym): bump the Gym submodule #5195 now has a commit for that (5ff3ddfd25).
  • test_recipes_and_test_suites.py and test_config_validation.py: 451 passed.
  • The prefix-recovery test on 2 GPUs, natively and through Dynamo (see Validation).

The other functional and cluster runs under Validation used an earlier pin (9aac609, the same dialect commits on the older Gym base) and were not repeated, except the multimodal comparison on vLLM 0.33, which ran on the current pin.

Validation

Step-order test (tests/unit/models/generation/test_gym_chat_path.py, 36 cases). Each sends one call down both paths: the native routes over ASGI, and our chat engine, a spawned render pool and the adapter. It checks they run the same prompt, media and SamplingParams, stage the same capture record, and return the same reply. The cases cover:

  • the capture decision: the cut holding and failing, and empty tool content with string and structured content;
  • splicing: an echoed prefix, a TQ prefix, and turns ending on a token that isn't the tokenizer's EOS;
  • an image after an echoed turn (Qwen2.5-VL);
  • score centering, Qwen3's and nano_v3's parsers, every sampling field, and refusals;
  • a call without max_tokens, a tool schema with a null default, and a named tool_choice under a parser without a structural tag;
  • measurement, including a history past the context window;
  • token information with top_logprobs omitted or null, served as 0 on both paths;
  • calls that do not finish, where both paths must also fail the capture with the same reason and tell Gym the same: a field vLLM refuses while the body is parsed, a spliced prompt that fills the context, a full queue, an out-of-vocabulary token, an engine error and a client disconnect;
  • reasoning_ended after the model's own earlier turn, and router replay's staged routes.

Each case fails when the code it covers is reverted.

Functional runs on 2× RTX PRO 6000. Each native SingleController test passed with ++policy.generation.dynamo=true and check_dynamo_overlay.sh. The first three rows ran at 45add8b0a1, the rest at 817107f3a0:

Test Result
grpo_dp_single_controller.sh pass; 394 KV events applied, gen_kl_error 0.0006
grpo_async_gym_single_controller.sh with capture pass; 253 KV events applied, gen_kl_error 0.0011
grpo_async_gym_single_controller_context_compaction.sh (forest training) pass; 3,719 KV events applied, gen_kl_error ≤ 0.0011. At 817107f3a0: of 128 chat calls, 20 candidates served token-in and 24 rooted new segments
grpo_async_gym_single_controller.sh without capture pass
grpo_dp_single_controller_reload_refit.sh pass
Gym capture + router replay with deferred routes pass; gen_kl_error 0.0003
nemotron_omni_clevr_vllm_token_capture_single_controller_1n2g.sh (VLM, media capture) test checks pass; media_row_rate 1, gen_kl_error 0.0004. The overlay check's KV-event line fails only because this config turns prefix caching off

Prefix recovery (at 26d9d01bbb):

  • Both paths, unit: test_gym_chat_path.py cuts a call with the worker's own cut, resumes it natively and through Dynamo, and compares both with the uninterrupted call: the prompt and SamplingParams the engine ran, the staged row, the reply's content, reasoning and tool calls, and usage. The cut lands inside the reasoning, after it, inside the tool call, and after the call finished; one case loses the cut's rows and regenerates.
  • feat(checkpoint): recover active vLLM generation prefixes #4506's shared restore cases run through the render worker's reply builder (test_dynamo_render_pool.py, 8 cases × 2 tokenizer styles).
  • The cut: test_dynamo_generation_cut.py fans a cut out to four engines (one holds a call, one holds none, one raises, one answers after the deadline). Against Dynamo 1.5.1 with the real frontend launcher and three stub engines, one of them slow: 404 before capture is set up, 401 for a wrong bearer, every engine sent the same deadline, an answer at that deadline, and a receipt that passes Gym's validate_for with only the slow engine's call failed.
  • Functional, 2 local GPUs: test(checkpoint): add Workplace prefix-recovery parity #4507's prefix-recovery test (8ca22f074e, not part of this PR) passes natively (504 s) and with ++policy.generation.dynamo=true (517 s): all 8 rollouts resumed, log probabilities bit-identical to the uninterrupted run, and check_dynamo_overlay.sh. Turn recovery behind Dynamo passes (265 s).
  • Rerun at 766f81d004, on Gym main and with the test now in the repo: it passes natively (1,291 s; the first run, which also rebuilt Gym's server venvs) and with ++policy.generation.dynamo=true (804 s), with the same checks. Behind Dynamo, stop_phase1 also stops the services that phase one's kill orphans, so the overlay's leak check passes.
  • Reverting any of the 23 lines these guard fails a test.

Rerun at 26d9d01bbb, after the shared steps landed: grpo_async_gym_single_controller.sh with capture passes natively (114 s) and through Dynamo (117 s; 251 KV events applied, gen_kl_error 0.0011), and grpo_dp_single_controller.sh through Dynamo (64 s; 393 KV events applied).

Unit tests at 26d9d01bbb, run without -x so one failure cannot hide the rest: vLLM lane 119 (the same 119 with nemo_gym not importable, as in CI's vLLM lane), vLLM+Gym lane 66 and Gym lane 1,203 passed. The default lane (4,371 passed) fails only locally: no nemo_automodel, and the Ray actors that leaves behind. CI's pre-commit hooks, run over every file this PR changes, and project-mode pyrefly are clean.

Merge with main (e6d892c552, which bumps vLLM from 0.29.0 to 0.33.0; the commit before #4583), at e449a14141. Run locally on a venv synced from the merged lock (vLLM 0.33.0, torch 2.14.0, ai-dynamo 1.5.1):

  • vLLM lane 128 passed, vLLM + Gym lane 64 passed, Gym lane 1,196 passed and the 14 code_gen errors, config tests 548 passed.
  • test_vllm_generation.py: 196 passed. The other 10 need nemo_automodel, which the vLLM venv cannot hold.
  • grpo_async_gym_single_controller.sh with capture through Dynamo on 2 GPUs passes, with check_dynamo_overlay.sh (gen_kl_error 0.0011, 224 KV events applied).
  • The other functional and cluster runs in this section ran on vLLM 0.29 and were not repeated at this merge. The multimodal comparison has since been repeated on vLLM 0.33 (under "Native vs Dynamo validation").

What the merge needed beyond its conflicts: main's two vLLM 0.33 changes to the chat handler are applied in chat_server.py, where this PR moves that code, and the nightly GPU-hour cap goes from 5310 to 5318, because main added an 8-hour nightly under the cap this PR's swe1 nightly had filled.

Merge with main again (bb9ed784e4, #4583), at 7e4fbd5ea4, on the same venv:

  • vLLM lane 128 passed, vLLM + Gym lane 69 passed, Gym lane 1,212 passed and the 14 code_gen errors, config tests 548 passed. test_vllm_generation.py as above.
  • feat(checkpoint): recover multimodal generation prefixes #4583's restored-image handler test passes through start_capture. Two new cases cover the Dynamo engine: a restored image call stages its reprocessed pixels, or is refused with retained_media_changed before it begins or generates; and a cut writes a media call's tensor digests into its first prefix row. With the restored-media call removed, feat(checkpoint): recover multimodal generation prefixes #4583's test and both restore cases fail.
  • The prefix-recovery test on 2 GPUs passes natively (445 s) and with ++policy.generation.dynamo=true (765 s): all 8 rollouts resumed, log probabilities bit-identical to the uninterrupted run, and check_dynamo_overlay.sh (1,662 KV events applied). feat(checkpoint): recover multimodal generation prefixes #4583 moves stop_phase1 into the shared _gym_prefix_recovery_phases.sh, and the Dynamo cleanup moved there with it.
  • Not run: feat(checkpoint): recover multimodal generation prefixes #4583's Omni image prefix-recovery test, natively or through Dynamo. Its checkpoint is not on this machine, and the test is in the GB200 Omni list only, so through Dynamo a restored image call is covered by the unit cases above.

Third merge with main (705953be86: #5074, several HTTP frontends per engine, and #5202, dynamic sampling with token capture), at 936f5d28c6; head 027b75a1f1, on the same venv:

  • vLLM lane 139 passed, vLLM + Gym lane 69 passed, Gym lane 1,213 passed and the 14 code_gen errors, config tests 550 passed. test_vllm_generation.py: 207 passed, with the same 10 that need nemo_automodel.
  • grpo_async_gym_single_controller.sh with capture on 2 GPUs passes through Dynamo (145 s; 82 requests routed, 218 KV events applied, gen_kl_error 0.0010) and natively with http_api_server_count=2 (205 s; gen_kl_error 0.0011). The second run puts feat(vllm): share one engine across multiple HTTP frontends #5074's spawned frontend on the chat server this PR moves to chat_server.py.
  • The prefix-recovery test passes with ++policy.generation.dynamo=true (589 s; 40 requests routed, 1,694 KV events applied) and natively (641 s): 8 rollouts resumed, log probabilities bit-identical over 9,312 tokens.

What the merge needed beyond its conflicts: the nightly GPU-hour cap is 5392, the merged list's total (main's 5296 plus 96 for the swe1 Dynamo nightly), and http_api_server_count above 1 is rejected with dynamo: true (above).

Fourth merge with main (aa0f77af51: #5206 group length penalties, #5036, #5226, #4784 dynamic sampling for the streaming samplers, #5239 checkpointing configs as BaseModel, and #5244), at 4581367100:

  • main changed no generation code, pyproject.toml, uv.lock or submodule pin, so only the tests the merge could affect were rerun, on the same venv: config tests 555 passed, and the three test files changed on both sides (test_setup.py, test_grpo.py, test_distillation.py) 861 passed.
  • The unit lanes and functional runs above were not repeated at this merge.

The one conflict was the nightly GPU-hour cap, now 5480, the merged list's total (main's 5384 plus 96 for the swe1 Dynamo nightly).

Fifth merge with main (8f3595a112: #5137 two-phase SimpleStorage checkpoint capture, and #5195 the Gym bump), at e3aec3d1ce:

  • No conflicts. The Gym pin stays 9e2e5d0ad0, which contains the pin build(gym): bump the Gym submodule #5195 brought to main.
  • Rerun on the same venv: config tests 459 passed, test_rollouts.py 107 passed, and the Gym-marked run of the four Gym test files 85 passed, including the 14 code_gen rollout cases that failed before this merge.
  • The other unit lanes and the functional runs were not repeated at this merge.

Native vs Dynamo validation (CMH, GB300)

Each config ran three ways with identical overrides: (a) main at 27fac5f3d6, native vLLM; (b) this stack with Dynamo off, at #5117 3bab36fee4; (c) this stack with policy.generation.dynamo=true, at #5117 8a5a90ef7a, which sits on this PR's 56774f269d. #5117 carries this PR and #5115, so both stack arms are on ai-dynamo 1.7.0.dev20261106, not the 1.5.1 this PR pins. The two stack commits differ only in the Dynamo frontend's chat workers. Runs: nvidia/nemo-rl-dynamo-parity.

The test scripts run unchanged, with more steps. The score-centering and Nano tests cannot train as written (one read-only tool, Qwen3-0.6B, the Nano Base checkpoint), so those three use the instruction-tuned model each config names, all 27 workplace-assistant tools, a 16,384-token context and 8 prompts × 16 rollouts per step, in every arm.

Forest training (context compaction), Qwen3-0.6B (C43 run 1; forest)

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 9 of 30 11 of 30 7 of 30
Reward, mean 0.1302 0.1302 0.1333
Turns per rollout, mean 2.012 2.178 1.932
Gradient norm, median 0.01575 0.0219 0.01485
gen_kl_error, median 0.0009504 0.0009318 0.0009384
token_mult_prob_error, median 1.023 1.022 1.023
Step time, median of steps 2+ (s) 27.21 30.7 29.99
Step time, step 1 (s) 46.71 49.29 78.54
Generation time, median of steps 2+ (s) 7.221 7.428 7.499
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 0

Score centering, Qwen3-30B-A3B-Instruct-2507 (C43 run 2; score-centering-instruct)

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 20 of 20 20 of 20 20 of 20
Reward, mean 0.4629 0.493 0.5055
Turns per rollout, mean 3.619 3.629 3.806
Gradient norm, median 0.7038 0.7207 0.6788
gen_kl_error, median 0.003708 0.003883 0.003813
token_mult_prob_error, median 1.029 1.027 1.032
Step time, median of steps 2+ (s) 39.68 44.94 40.94
Step time, step 1 (s) 78.83 85.39 106.1
Generation time, median of steps 2+ (s) 11.03 10.61 12.29
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 2

Score centering with filtered sampling and sampler replay, Qwen3-30B-A3B-Instruct-2507 (C43 run 2; score-centering-filtered-instruct)

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 20 of 20 20 of 20 20 of 20
Reward, mean 0.5137 0.4949 0.5102
Turns per rollout, mean 3.808 3.652 3.888
Gradient norm, median 0.5218 0.5315 0.4798
gen_kl_error, median 0.002719 0.002566 0.002817
token_mult_prob_error, median 1.021 1.018 1.022
Step time, median of steps 2+ (s) 44.88 43.47 50.2
Step time, step 1 (s) 93.55 91.63 119.1
Generation time, median of steps 2+ (s) 11.24 11.4 12.4
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 5

Multimodal (Omni CLEVR, media token capture) (C43 run 3; omni-vlm)

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 0 of 16 2 of 16 1 of 16
Reward, mean 1 0.9922 0.9961
Turns per rollout, mean 1 1 1
Gradient norm, median 0 0 0
gen_kl_error, median 0.001549 0.001703 0.002164
token_mult_prob_error, median 1.017 1.019 1.021
Step time, median of steps 2+ (s) 12.38 11.84 11.48
Step time, step 1 (s) 133.1 139.3 189.8
Generation time, median of steps 2+ (s) 0.001165 0.001102 0.001128
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 0

Multimodal, repeated on vLLM 0.33 (omni-vlm, at 027b75a1f1)

The table above ran on vLLM 0.29. This repeats it after the merges with main, with the same test, overrides, cluster and node shape: (a) main at 705953be86, the main commit this PR contained at 027b75a1f1; (b) this PR at 027b75a1f1 with Dynamo off; (c) the same commit with policy.generation.dynamo=true. All three are on vLLM 0.33.0. Arm (c) is on ai-dynamo 1.5.1, the version this PR pins, where the other tables in this section used the 1.7 nightly.

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 5 of 16 3 of 16 4 of 16
Reward, mean 0.9805 0.9883 0.9844
Turns per rollout, mean 1 1 1
Gradient norm, median 0 0 0
gen_kl_error, median 0.00187 0.001868 0.001775
token_mult_prob_error, median 1.019 1.019 1.018
Step time, median of steps 2+ (s) 10.41 11.25 12.11
Step time, step 1 (s) 156.6 160.4 150.1
Generation time, median of steps 2+ (s) 0.001025 0.001141 0.001106
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 0

All three runs train 16 steps and pass every gate the test defines. In arm (c) the frontend routed all 256 requests, and no frontend, etcd or NATS process was left on the node.

  • gen_kl_error: the Dynamo run's median was the highest of the three on vLLM 0.29 and is the lowest here.
  • First step: the Dynamo run's is no longer slower than native (150 s against 157 and 160 s; on vLLM 0.29 it was 190 s against 133 and 139 s). Since e1c510f8dc the frontend's chat workers are built while the engines load, and the first chat call waited 0.0 s for them.
  • Later steps: the Dynamo run's median step is 12.1 s against 10.4 and 11.3 s. The difference is on the trainer side (policy_training 8.1 s against 6.9 and 7.6 s), not in generation or weight sync, and the order is the reverse in the table above. With one run per arm it cannot be told apart from node-to-node spread.
  • Setup: the runs use the nightly image built from main (bb9ed784e4), whose uv.lock is identical to 705953be86's. This PR pins a newer Gym that requires openai 2.45.0, so arms (b) and (c) put that openai on PYTHONPATH and build their own Gym server venvs. An image built from this PR's lock would need neither.
  • Locally: the same test at 027b75a1f1 also passes on 2 GPUs, natively and through Dynamo, with vLLM given 85% of a GPU (the model does not fit in the YAML's 60% of a 96 GB card).

Nano 30B instruct with Gym: tool and reasoning parsers, token capture, deferred R3 (C22; nano-gym-instruct)

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 20 of 20 20 of 20 20 of 20
Reward, mean 0.4238 0.4359 0.4398
Turns per rollout, mean 1.891 1.926 1.925
Gradient norm, median 0.4488 0.4714 0.4766
gen_kl_error, median 0.0009219 0.0009119 0.0009541
token_mult_prob_error, median 1.017 1.018 1.018
Step time, median of steps 2+ (s) 49.94 47.97 48.49
Step time, step 1 (s) 222.1 219 235.2
Generation time, median of steps 2+ (s) 0.01417 0.01522 0.01483
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 0

R3 on text rollouts (PPO, Qwen3-30B-A3B) (extra; r3-ppo)

(a) main (b) Dynamo off (c) Dynamo on
Steps with a non-zero advantage 20 of 20 20 of 20 20 of 20
Reward, mean 0.02734 0.02266 0.02187
Turns per rollout, mean 1 1 1
Gradient norm, median 0.3792 0.3768 0.3764
gen_kl_error, median 0.0007807 0.000782 0.0007786
token_mult_prob_error, median 1.013 1.013 1.013
Step time, median of steps 2+ (s) 111.8 109.6 112
Step time, step 1 (s) 289.2 283.9 281.9
Generation time, median of steps 2+ (s) 0.006009 0.005474 0.005882
Failed or dropped Gym calls 0 0 0
Gym rows redispatched 0 0 0

Every gate each test defines passes in all three arms, with two exceptions that are the same in every arm: gates that count steps (these runs are longer), and the filtered score-centering test's max(token_mult_prob_error) < 1.05 (see the third note). The Dynamo arm's routing check (check_dynamo_overlay.sh) confirms requests went through the frontend in every config.

Four notes:

  • First step. With Dynamo on, the first step of a Gym config is 13 to 57 s slower than native. Before 56774f269d it was 60 to 130 s. On this cluster the render pool's first worker takes 67 to 86 s to start, longer than the time between engine registration and the first chat call. Later steps match, and text rollouts show no delay. e1c510f8dc builds the workers while the engines load instead. Only the multimodal config has been rerun with it on this cluster (the vLLM 0.33 table above), and there the Dynamo run's first step is no slower than native.
  • A rollout that fills its context. In the Qwen3-30B runs, 1 to 8 of 2,560 rollouts per run reach the 16,384-token limit, in every arm. Native ends the rollout there and keeps it. With Dynamo on, the frontend (ai-dynamo 1.7) answers the refused call with {"message":"Bad Request"} in place of the chat engine's message and capture acknowledgement, so Gym fails the call and the row is redispatched (2 and 5 rows above).
  • Logprob-error spikes. On the two Qwen3-30B score-centering configs, token_mult_prob_error jumps on a few steps in every arm, main included, while the median stays near 1.02. Steps above 1.1, of 40 per arm: 5 on main, 5 with Dynamo off, 6 with Dynamo on. The Nano config, which replays routed experts, has none.
  • Training. Both score-centering configs, Nano and R3 PPO have non-zero advantages on all 20 steps in every arm. Forest training does on about a third of its steps, and the multimodal test hardly at all (its fixture is one image).

Not covered here: RLVR at very high concurrency (C43 run 4), a subagent config (none exists in the repo), and the swe1 nightly itself (legacy path; the Nano config above covers what it tests).

Before merge

  • The swe1 nightly with Dynamo on and off, with its metric-check table (C22). It has not been run. The Nano 30B run under Validation covers its tool and reasoning parsers, token capture and deferred R3, and the tables there give Gym step and generation times, native vs Dynamo.
  • RLVR at very high concurrency, on main and on this PR with Dynamo off and on (C43 run 4). The other three runs are under Validation; forest training ran without subagents, since no config in the repo sets them up. Still to report from a Dynamo-on run: frontend chat calls/s, render-pool queue wait (the agentic one on more engines than render_workers) and the frontend's TQ prefix-read rate, which one prefix cache now serves for the whole fleet. The runs above do not log them.
  • build(gym): bump the Gym submodule #5195 is on main and merged into this branch, so the code_gen rollout test passes again (see Merge order). The Gym re-pin (C32) is done.

Follow-ups

  • Cuts at scale: every engine receives the whole inventory. At 100K+ calls in flight the cut should go only to the engine that holds each call.
  • Several frontends (feat(dynamo)!: Dynamo 1.7 nightly, high session cap, frontends across nodes #5115): each needs its own cut port, listed in the order of Gym's base URLs.
  • Megatron halves of the shared chat path.
  • Image prefix recovery behind Dynamo, end to end: feat(checkpoint): recover multimodal generation prefixes #4583's Omni test with ++policy.generation.dynamo=true. It is not in the Dynamo L1 list, and a restored image call through Dynamo has unit coverage only.
  • Pixels off the request: send processed media over Dynamo's shared-memory or NIXL transfer instead of inside the request.
  • VllmRequestCapture: move the vLLM worker's capture lifecycle into its own object, as Megatron has, so the shared steps stop calling the worker's private capture members.
  • reasoning_ended for a resumed call: a call with a structural-tag tool parser that was cut after its own </think> is told its reasoning has not ended, so the tag is not applied to the resumed tokens. Reading it from the template's ids plus the call's saved tokens needs the reasoning parser on the worker behind Dynamo.
  • Session affinity: the frontend now splices before routing, so the KV router sees a capture session's prefix. Whether to keep session_affinity_ttl_secs on by default needs measuring at scale, on Dynamo 1.7.

🤖 Generated with Claude Code

jthomson04 and others added 14 commits October 24, 2026 13:04
…end SDK

`policy.generation.dynamo: true` on a vLLM generation config is now the whole
Dynamo switch. NeMo RL's own vLLM workers register their AsyncLLM with a
managed dynamo.frontend through Dynamo's engine-agnostic backend SDK
(dynamo.common.backend), from inside each worker's actor loop. No Dynamo-owned
vLLM code runs, so refit, pause, quantization, speculative decoding and
parallelism are the native backend's own, and a vLLM bump never waits on
Dynamo.

- One request pipeline (vllm/request_pipeline.py) serves native
  generate_async, the native OpenAI server and the Dynamo adapter: the
  sampling-profile check, clamps, R3, token fields, capture, prefix splice
  and media.
- Rollouts go through the frontend's KV-aware router. The engine publishes
  KV events, and functional overlays fail if the router applies none.
- NeMo-Gym calls go through the frontend in Gym's Dynamo dialect
  (NVIDIA-NeMo/Gym#4817), with engine-side token and media capture.
- Startup rejects what Dynamo cannot serve: colocated generation, fleet
  health, the generation router, a bearer token, R3 with Gym but no capture,
  parser plugins and chat settings with no frontend counterpart.
- Removed: backend: dynamo, DynamoGeneration, the dynamo.vllm launcher, its
  refit routes, token wrapper, metrics scraper and isolated venv. Old keys are
  rejected with the migration.
- ai-dynamo and ai-dynamo-runtime 1.5.1 join the vllm extra, without the
  planner deps. BUILD_DYNAMO installs only etcd and nats-server.
- CI: the Dynamo lane runs native SingleController tests as overlays: math,
  Gym token capture, reload refit, and R3 with deferred-route capture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Run and CI breaks:
- Each engine publishes KV events on its own abstract ipc:// socket, so
  engines on one node can no longer draw the same port (C1).
- Engines get an OS-assigned DYN_SYSTEM_PORT, so a frontend on another node
  can read a custom chat template (C2).
- The Gym capture test skips without nemo_gym (C3).
- Gym calls apply the generation config's max_tokens only when the request
  leaves it unset, as vLLM's server does (C4).
- Lint: minimized swe1 recipe (C5) and import order (C6).

Behavior:
- vLLM client errors reach the client as InvalidArgument with their message,
  not as a bare 500 (C7).
- Non-capture Gym calls return routed experts with R3 on, and Gym#4817
  (now a369e28) lifts them, so R3 + Gym no longer needs token capture (C8).
- A disconnected capture call fails instead of being staged (C9).
- EP > TP is rejected at startup with Dynamo (C10).
- The KV-event block size matches vLLM's main-attention kinds (C17).

Messages and docs: timeout wording and error (C11, C12), missing etcd/NATS
advice (C13), dependency bounds (C19) and ray.sub's port layout (C20).

Tests: a Gym-without-capture Dynamo overlay in L1 (C14), the rollout retry
loop (C15), media dispatch and failure mapping (C16), hybrid block size
(C17) and the Gym engine_data key contract (C18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolves the docker/Dockerfile conflict with NVIDIA-NeMo#4340 (uv cache mounts). The
BUILD_DYNAMO RUN keeps this branch's form: docker/dynamo/install.sh now only
downloads etcd and nats-server, so it is not a uv-writing RUN and takes no
uv cache mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Gym#4817 (jthomson04/dynamo-frontend-dialect) is rebased onto Gym main
and now sits on top of it; its head moves from a369e28 to a5f936c. The
only conflict was Gym main's new custom-tool-strategy validator, which
sits next to the Dynamo frontend validator; both are kept. Gym's
dependencies are unchanged, so uv.lock needs no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…osition

From the NVIDIA-NeMo#4905 review:
- C23: setup_nemo_gym_config wrote the dialect as a top-level policy_model
  overlay, which a sharded Gym plan rejects as unowned. Setup now stashes it
  under a NeMo RL key (dynamo_dialect, alongside tokenizer_config), and
  _build_gym_actor_config applies it to each actor's composed config. A
  shard's own template kwargs still win over the native defaults.
- C24: the local install instructions export NEMO_RL_DYNAMO_BIN_DIR, so the
  run that follows finds etcd and nats-server.
- C25: the two in-function rationale blocks in native_engine.py are one line
  each; the design doc carries the full reasons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve examples/configs/grpo_math_1B.yaml: keep main's new
lora_refit_mode default and this branch's comment, which no longer lists
the removed dynamo backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve tests/unit/algorithms/test_grpo.py: main (NVIDIA-NeMo#4712) added Gym
log-dir assertions to test_setup_initializes_noncolocated_dynamo_with_nemo_gym,
which this branch deletes with backend: dynamo. The log-dir fix is not
Dynamo-specific, so its assertions move to the surviving Gym setup test,
test_setup_starts_nemo_gym_for_trtllm, with the same parametrization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve the configure_generation_config conflict: keep this branch's
vllm-only block (backend: dynamo is removed) with NVIDIA-NeMo#4554's reworded
load_format comment. NVIDIA-NeMo#4554 also makes setup_nemo_gym_config read
config.data, so give the two Dynamo dialect tests' configs data={} as
upstream did for its own tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolves the conflict in nemo_rl/utils/routed_experts_codec.py: this branch
added routed_experts_envelope_dtype and NVIDIA-NeMo#4915 added slice_routed_experts at
the same spot. Both are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…helpers

Split generate_async's per-sample work into building the result
(request_pipeline.rollout_result: sampled logprobs, router-replay route
alignment and its fallback warning) and assembling the output tensors
(rollout_output). The rollout token budget and SamplingParams become
worker methods (_rollout_max_new_tokens, _rollout_sampling_params).

vllm/utils gains off_policy_sampling_error (the chat endpoint's on-policy
checks, unchanged) and chat_routed_experts / chat_generation_tokens (the
loop bodies of the attach_* response helpers).

No behaviour change for native vLLM: the new generate_async wiring tests
pass against both the old and the new implementation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…engine

Stacks this PR on the vLLM request-pipeline refactor split out at review
request; brings in upstream main 0b305c9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
At review request, NeMo-Gym through the Dynamo frontend moves to a
follow-up PR that waits on NVIDIA-NeMo/Gym#4817, so this PR no longer
needs a Gym bump:

- Gym's submodule is back on main's pin.
- The engine's Gym path (_generate_gym, capture staging, prefix splice),
  the Gym dialect setup, session affinity, client_base_urls and the
  Gym-only startup checks move to the follow-up.
- NeMo-Gym with policy.generation.dynamo=true is refused at startup in
  should_use_nemo_gym, before any cluster starts.
- The Dynamo L1 keeps its SingleController math and reload-refit overlays;
  the swe1 Dynamo + Gym nightly is disabled until the follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Restores NeMo-Gym support under policy.generation.dynamo=true, split out of
the Dynamo core PR at review request:

- Gym's policy model calls the frontend in Gym's Dynamo dialect
  (NVIDIA-NeMo/Gym#4817); the engine serves those calls as the native chat
  endpoint does: on-policy sampling check, echoed-prefix splice, context
  clamp, token fields and routed experts in engine_data.
- Token capture stages from the engine process (dynamo_worker backend),
  with frontend session affinity keeping a rollout on one engine.
- The Dynamo L1 regains its three Gym overlays, and the swe1 Dynamo + Gym
  nightly is re-enabled.
- Gym is pinned to Gym#4817's rebased head b0e681c; uv.lock records Gym's
  updated requirements (no resolved versions change).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04 jthomson04 added the CI:L1 Run doctests, unit tests, and functional tests label Oct 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 3f45420

jthomson04 added a commit to jthomson04/RL that referenced this pull request Oct 29, 2026
Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4905's split (vLLM refactor NVIDIA-NeMo#4978, Dynamo core
NVIDIA-NeMo#4905, NeMo-Gym through Dynamo NVIDIA-NeMo#4979) and the Gym#4817/#5161 rebase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
copy-pr-bot Bot pushed a commit that referenced this pull request Oct 29, 2026
#4905 was split at review request: the vLLM refactor is #4978, #4905 is
now Dynamo core, and NeMo-Gym through Dynamo is #4979. This PR now stacks
on #4979. Gym is re-pinned to Gym#5161's head aa808b7, rebased onto Gym
main 5911d628c on top of Gym#4817's b0e681c.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions github-actions Bot added Documentation Improvements or additions to documentation CI Relating to CI labels Oct 29, 2026
jthomson04 and others added 3 commits October 29, 2026 13:47
Address review on this PR:

- _serve_rollout runs a rollout's whole sequence (budget, nothing-fits,
  SamplingParams, generate, prompt check, result), so every host that
  serves a rollout from this engine calls one method instead of copying
  the steps. It replaces _rollout_sampling_params.
- generate_async asserts a single sample, so it no longer fans out
  per-sample tasks: it formats the prompt, calls _serve_rollout and
  yields the output. vLLM still aborts the request on cancellation.
- rollout_output drops pad_token_id, whose fill never reached the output.
- off_policy_sampling_error documents that vLLM fills unset top_p/top_k
  from generation_config.json, so callers must pin top_k=-1.
- chat_routed_experts drops a dead isinstance check.
- Tests: a route test for the on-policy assert and the top_k pin; the
  budget tests fold into one parametrization that covers a full context
  with speculative decoding; the wiring test asserts the exact output
  keys and a missing logprob; the R3 pipeline test uses vLLM 0.29's route
  shape. Two pipeline tests the wiring tests now cover are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…engine

Brings in NVIDIA-NeMo#4978's review fixes. The engine's rollout path now calls the
worker's _serve_rollout instead of re-typing its steps, and the driver
drops rollout_output's removed pad_token_id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Brings in NVIDIA-NeMo#4978's review fixes through NVIDIA-NeMo#4905: the engine's rollout path
calls the worker's _serve_rollout. The Gym path is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 2cfb157 (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR serves NeMo-Gym through the Dynamo frontend. It is stacked on #4905 (Dynamo core), which sits on #4978 (the vLLM refactor), so this review covers only this PR's own range, 1a2a581ee8..2cfb157e9a. That range also moves the Gym pin from 3f6f2059 to b0e681c: Gym#4817 plus about 260 Gym-main commits. Gym changes in that range break two CI jobs here, and both have small fixes:

  1. (C1) The L0 Nemo_Gym unit lane fails: 121 data-plane tests and one captured-media case still use Gym's old rollout_id name and staging schema 2. A 3-file test fix restores the lane.
  2. (C2) The GB200 distillation nightly fails at config load, because Gym dropped the alias that resolved acereason-math.yaml. A one-line YAML change fixes it.

The other comments:

  1. (C3) Please fix in this PR: with the 1800 s affinity TTL, Dynamo's 65,536-session table fills at about 36 new captured rollouts per second, and new rollouts then get HTTP 529. No shipped config reaches that rate today. Lower the default to 300.
  2. (C4) The disconnect branch and the media is None guard in _generate_gym have no test; a 2-file diff adds both.
  3. (C5) "ng_commit_coords" is an untested copy of a Gym constant; import the constant where it is read.
  4. (C6) The dialect test should read Gym's field names back through Gym's own config class, which drops unknown keys.
  5. (C7) Low: an output that vLLM ended with finish_reason == "error" is staged before the client gets its 500; raise first, as vLLM's chat server does.
  6. (C8) Low: the new capture test and 8 production sites still use Gym's deprecated rollout_id; rename them in the same commit as C1.
  7. (C9) Low: give vLLM the request's skip_special_tokens, so that vLLM and the frontend stop at the same place.
  8. (C10) Low: reject extra_body.bad_words at setup, since the frontend answers every such request with a 400.
  9. (C11) Doc: template kwargs sent through extra_body are dropped under Dynamo.
  10. (C12) Doc: the frontend refuses more than 4 stop strings or stop token IDs with a 500.
  11. (C13) Gym sends media inline; raise the frontend's 45 MB body limit and the 32 MiB request-plane limit.
  12. (C14) Doc: request_timeout_s: 7200 in the swe1 recipe bounds nothing now, and the exemplar comment needs the same Gym note as config.py.
  13. (C15) Nit: the _apply_dynamo_dialect docstring says it does not mutate its argument, but it does.

Notes with no inline comment

  • Follow-up to carry over. #4905's review asked to move the vLLM worker's token-capture lifecycle into its own object (VllmRequestCapture). That code now lives in this PR: the Dynamo engine (NativeVllmEngine) calls 9 private members of the vLLM worker, 8 of them added here. Please list it under this PR's Follow-ups.
  • Splice coverage. #4905's C14 overlay reaches the engine_data token arrays but not the ng_prefix splice (0 of 80 rollouts made a second call; turns_per_sample counts one per model call). A failed splice raises an AssertionError, which becomes a 5xx that is retried and absorbed by the rollout-failure budget, and nothing asserts on it. So a frontend or template-engine bump can break the splice without turning L1 red. (The EOS-count derivation itself matches native; see below.)
  • swe1 nightly. The re-enabled swe1 nightly (nightly.txt#L278) runs the legacy run_grpo_nemo_gym.py (.sh#L32). It is the first multi-engine Gym + Dynamo run, and no validation row covers it. No run is requested.
  • Nit. The comments at run_grpo_nemo_gym.py#L214 and run_distillation_nemo_gym.py#L114-L115 still name dp_openai_server_base_urls; Gym now gets client_base_urls.

What holds up

  • The split from #4905 lost nothing: the head tree equals the pre-split 89469caa2d except for the Gym pin and 2 uv.lock lines.
  • All 27 review threads on the stack still held at 3f45420d1f, except G3, which is deferred and listed in this PR's Follow-ups.
  • The EOS-count derivation for the echoed prefix matches native on real Qwen3 and Nemotron-3-Nano templates.
  • The disconnect branch labels its abort client_disconnect, which is right.

Generated by Claude Code

Comment thread 3rdparty/Gym-workspace/Gym Resolved
Comment thread 3rdparty/Gym-workspace/Gym Resolved
Comment thread docs/guides/dynamo-generation.md Resolved
Comment thread docs/guides/dynamo-generation.md Resolved
Comment thread nemo_rl/environments/nemo_gym.py Resolved
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 7e4fbd5 (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

jthomson04 and others added 3 commits November 14, 2026 16:28
…ing with token capture) into jthomson04/dynamo-gym

Conflicts:
- nemo_rl/models/generation/vllm/vllm_worker_async.py: imports keep both
  sides (main's VllmConfig and MultiAPIServerPool, this branch's
  VllmProgressReader; the chat-reply helpers stay in chat_server.py). The
  engine is built as on main, through the HTTP frontend pool when
  http_api_server_count > 1, after this branch sends the engine's chat
  settings to the Dynamo frontend.
- tests/unit/test_recipes_and_test_suites.py: the nightly GPU-hour cap is
  5392. The merged list totals 5392: main's 5296 plus 96 for the swe1
  Dynamo nightly.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
vllm_cfg.http_api_server_count (NVIDIA-NeMo#5074) adds HTTP frontend processes to each
engine's native server. With policy.generation.dynamo=true NeMo-Gym's calls
go through the Dynamo frontend, so those processes would serve none of them.
Startup now rejects the combination and points at
dynamo_cfg.frontend_args.render_workers. The native path is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
A run_test line without `fast` is skipped in every CI lane that reuses a
pre-built image, so CI would not show whether this test passes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 15, 2026
… prefix recovery) into jthomson04/dynamo-1.7

Three files conflicted:

- uv.lock: re-resolved with uv 0.11.28, starting from NVIDIA-NeMo#4979's lock. Against
  that lock it moves ai-dynamo and ai-dynamo-runtime from 1.5.1 to
  1.7.0.dev20261106, adds redis 8.1.0, and drops pydantic 2.13.0 and
  pydantic-core 2.46.0. Only ai-dynamo-runtime 1.5.1 pinned those two, so the
  lock no longer forks on them; most of the changed lines are that fork's
  markers going away.
- docs/guides/dynamo-generation.md: this branch's wording for several
  frontends, with its sentences unwrapped as main's docs now are.
- tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh:
  NVIDIA-NeMo#4979's side, which sources main's shared phase helper.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 15, 2026
NVIDIA-NeMo#4979 stops the Dynamo services that phase one's kill orphans from
stop_phase1, which main has since moved into
_gym_prefix_recovery_phases.sh. On this branch the services stop when
their starter is killed (d044de8), so nothing is left to stop: the
helper and the test are main's again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 15, 2026
… engine and dynamic sampling under token capture) into jthomson04/dynamo-1.7

No conflicts. uv.lock is unchanged: the merged commits change no dependency.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 15, 2026
…d multimodal prefix recovery) into jthomson04/gym-actor-concurrency

No conflicts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 027b75a (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 027b75a

terrykong
terrykong previously approved these changes Nov 15, 2026

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The remaining validation asks are accepted without their runs: the swe1 Dynamo nightly (C22), RLVR at very high concurrency (C43 run 4) and the frontend numbers (C73). See the replies on those threads. Please watch the swe1 Dynamo nightly on main after merge.

Comment thread tests/test_suites/nightly.txt Resolved
… streaming samplers, checkpointing configs as BaseModel) into jthomson04/dynamo-gym

Conflicts:
- tests/unit/test_recipes_and_test_suites.py: the nightly GPU-hour cap is
  5480. The merged list totals 5480: main's 5384 plus 96 for the swe1
  Dynamo nightly.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 4581367 (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

1 similar comment
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 4581367 (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

terrykong
terrykong previously approved these changes Nov 16, 2026
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 16, 2026
…namic sampling for the streaming samplers and checkpointing configs as BaseModel) into jthomson04/dynamo-1.7

No conflicts. uv.lock is unchanged: the merged commits change no dependency.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

jthomson04 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

#5195 is merged, and main with it is merged into this branch at e3aec3d1ce. This PR's diff no longer includes #5195's changes, so the earlier "do not merge until #5195" hold no longer applies.

@terrykong
terrykong disabled auto-merge November 16, 2026 21:48
… two-phase SimpleStorage checkpoint capture) into jthomson04/dynamo-gym

No conflicts. main now has NVIDIA-NeMo#5195, so this branch's diff against main no
longer includes it, and it brings NVIDIA-NeMo#5195's code_gen fix for the Gym rollout
test.

The Gym pin stays 9e2e5d0ad0, which contains main's 23557b7c2a. uv.lock and
pyproject.toml are unchanged and equal main's.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: e3aec3d (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

1 similar comment
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: e3aec3d (PR #4979 from jthomson04/dynamo-gym)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test e3aec3d

@terrykong
terrykong enabled auto-merge (squash) November 17, 2026 00:38
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 17, 2026
… two-phase SimpleStorage checkpoint capture) into jthomson04/dynamo-1.7

No conflicts. uv.lock is unchanged: the merged commits change no dependency.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants