Repository navigation
feat(dynamo): serve NeMo-Gym through the Dynamo frontend - #4979
jthomson04 wants to merge 125 commits into
Conversation
…end SDK `policy.generation.dynamo: true` on a vLLM generation config is now the whole Dynamo switch. NeMo RL's own vLLM workers register their AsyncLLM with a managed dynamo.frontend through Dynamo's engine-agnostic backend SDK (dynamo.common.backend), from inside each worker's actor loop. No Dynamo-owned vLLM code runs, so refit, pause, quantization, speculative decoding and parallelism are the native backend's own, and a vLLM bump never waits on Dynamo. - One request pipeline (vllm/request_pipeline.py) serves native generate_async, the native OpenAI server and the Dynamo adapter: the sampling-profile check, clamps, R3, token fields, capture, prefix splice and media. - Rollouts go through the frontend's KV-aware router. The engine publishes KV events, and functional overlays fail if the router applies none. - NeMo-Gym calls go through the frontend in Gym's Dynamo dialect (NVIDIA-NeMo/Gym#4817), with engine-side token and media capture. - Startup rejects what Dynamo cannot serve: colocated generation, fleet health, the generation router, a bearer token, R3 with Gym but no capture, parser plugins and chat settings with no frontend counterpart. - Removed: backend: dynamo, DynamoGeneration, the dynamo.vllm launcher, its refit routes, token wrapper, metrics scraper and isolated venv. Old keys are rejected with the migration. - ai-dynamo and ai-dynamo-runtime 1.5.1 join the vllm extra, without the planner deps. BUILD_DYNAMO installs only etcd and nats-server. - CI: the Dynamo lane runs native SingleController tests as overlays: math, Gym token capture, reload refit, and R3 with deferred-route capture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Run and CI breaks: - Each engine publishes KV events on its own abstract ipc:// socket, so engines on one node can no longer draw the same port (C1). - Engines get an OS-assigned DYN_SYSTEM_PORT, so a frontend on another node can read a custom chat template (C2). - The Gym capture test skips without nemo_gym (C3). - Gym calls apply the generation config's max_tokens only when the request leaves it unset, as vLLM's server does (C4). - Lint: minimized swe1 recipe (C5) and import order (C6). Behavior: - vLLM client errors reach the client as InvalidArgument with their message, not as a bare 500 (C7). - Non-capture Gym calls return routed experts with R3 on, and Gym#4817 (now a369e28) lifts them, so R3 + Gym no longer needs token capture (C8). - A disconnected capture call fails instead of being staged (C9). - EP > TP is rejected at startup with Dynamo (C10). - The KV-event block size matches vLLM's main-attention kinds (C17). Messages and docs: timeout wording and error (C11, C12), missing etcd/NATS advice (C13), dependency bounds (C19) and ray.sub's port layout (C20). Tests: a Gym-without-capture Dynamo overlay in L1 (C14), the rollout retry loop (C15), media dispatch and failure mapping (C16), hybrid block size (C17) and the Gym engine_data key contract (C18). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolves the docker/Dockerfile conflict with NVIDIA-NeMo#4340 (uv cache mounts). The BUILD_DYNAMO RUN keeps this branch's form: docker/dynamo/install.sh now only downloads etcd and nats-server, so it is not a uv-writing RUN and takes no uv cache mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Gym#4817 (jthomson04/dynamo-frontend-dialect) is rebased onto Gym main and now sits on top of it; its head moves from a369e28 to a5f936c. The only conflict was Gym main's new custom-tool-strategy validator, which sits next to the Dynamo frontend validator; both are kept. Gym's dependencies are unchanged, so uv.lock needs no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…osition From the NVIDIA-NeMo#4905 review: - C23: setup_nemo_gym_config wrote the dialect as a top-level policy_model overlay, which a sharded Gym plan rejects as unowned. Setup now stashes it under a NeMo RL key (dynamo_dialect, alongside tokenizer_config), and _build_gym_actor_config applies it to each actor's composed config. A shard's own template kwargs still win over the native defaults. - C24: the local install instructions export NEMO_RL_DYNAMO_BIN_DIR, so the run that follows finds etcd and nats-server. - C25: the two in-function rationale blocks in native_engine.py are one line each; the design doc carries the full reasons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve examples/configs/grpo_math_1B.yaml: keep main's new lora_refit_mode default and this branch's comment, which no longer lists the removed dynamo backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve tests/unit/algorithms/test_grpo.py: main (NVIDIA-NeMo#4712) added Gym log-dir assertions to test_setup_initializes_noncolocated_dynamo_with_nemo_gym, which this branch deletes with backend: dynamo. The log-dir fix is not Dynamo-specific, so its assertions move to the surviving Gym setup test, test_setup_starts_nemo_gym_for_trtllm, with the same parametrization. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve the configure_generation_config conflict: keep this branch's vllm-only block (backend: dynamo is removed) with NVIDIA-NeMo#4554's reworded load_format comment. NVIDIA-NeMo#4554 also makes setup_nemo_gym_config read config.data, so give the two Dynamo dialect tests' configs data={} as upstream did for its own tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolves the conflict in nemo_rl/utils/routed_experts_codec.py: this branch added routed_experts_envelope_dtype and NVIDIA-NeMo#4915 added slice_routed_experts at the same spot. Both are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…helpers Split generate_async's per-sample work into building the result (request_pipeline.rollout_result: sampled logprobs, router-replay route alignment and its fallback warning) and assembling the output tensors (rollout_output). The rollout token budget and SamplingParams become worker methods (_rollout_max_new_tokens, _rollout_sampling_params). vllm/utils gains off_policy_sampling_error (the chat endpoint's on-policy checks, unchanged) and chat_routed_experts / chat_generation_tokens (the loop bodies of the attach_* response helpers). No behaviour change for native vLLM: the new generate_async wiring tests pass against both the old and the new implementation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…engine Stacks this PR on the vLLM request-pipeline refactor split out at review request; brings in upstream main 0b305c9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
At review request, NeMo-Gym through the Dynamo frontend moves to a follow-up PR that waits on NVIDIA-NeMo/Gym#4817, so this PR no longer needs a Gym bump: - Gym's submodule is back on main's pin. - The engine's Gym path (_generate_gym, capture staging, prefix splice), the Gym dialect setup, session affinity, client_base_urls and the Gym-only startup checks move to the follow-up. - NeMo-Gym with policy.generation.dynamo=true is refused at startup in should_use_nemo_gym, before any cluster starts. - The Dynamo L1 keeps its SingleController math and reload-refit overlays; the swe1 Dynamo + Gym nightly is disabled until the follow-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Restores NeMo-Gym support under policy.generation.dynamo=true, split out of the Dynamo core PR at review request: - Gym's policy model calls the frontend in Gym's Dynamo dialect (NVIDIA-NeMo/Gym#4817); the engine serves those calls as the native chat endpoint does: on-policy sampling check, echoed-prefix splice, context clamp, token fields and routed experts in engine_data. - Token capture stages from the engine process (dynamo_worker backend), with frontend session affinity keeping a rollout on one engine. - The Dynamo L1 regains its three Gym overlays, and the swe1 Dynamo + Gym nightly is re-enabled. - Gym is pinned to Gym#4817's rebased head b0e681c; uv.lock records Gym's updated requirements (no resolved versions change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test 3f45420 |
Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4905's split (vLLM refactor NVIDIA-NeMo#4978, Dynamo core NVIDIA-NeMo#4905, NeMo-Gym through Dynamo NVIDIA-NeMo#4979) and the Gym#4817/#5161 rebase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
#4905 was split at review request: the vLLM refactor is #4978, #4905 is now Dynamo core, and NeMo-Gym through Dynamo is #4979. This PR now stacks on #4979. Gym is re-pinned to Gym#5161's head aa808b7, rebased onto Gym main 5911d628c on top of Gym#4817's b0e681c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Address review on this PR: - _serve_rollout runs a rollout's whole sequence (budget, nothing-fits, SamplingParams, generate, prompt check, result), so every host that serves a rollout from this engine calls one method instead of copying the steps. It replaces _rollout_sampling_params. - generate_async asserts a single sample, so it no longer fans out per-sample tasks: it formats the prompt, calls _serve_rollout and yields the output. vLLM still aborts the request on cancellation. - rollout_output drops pad_token_id, whose fill never reached the output. - off_policy_sampling_error documents that vLLM fills unset top_p/top_k from generation_config.json, so callers must pin top_k=-1. - chat_routed_experts drops a dead isinstance check. - Tests: a route test for the on-policy assert and the top_k pin; the budget tests fold into one parametrization that covers a full context with speculative decoding; the wiring test asserts the exact output keys and a missing logprob; the R3 pipeline test uses vLLM 0.29's route shape. Two pipeline tests the wiring tests now cover are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…engine Brings in NVIDIA-NeMo#4978's review fixes. The engine's rollout path now calls the worker's _serve_rollout instead of re-typing its steps, and the driver drops rollout_output's removed pad_token_id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Brings in NVIDIA-NeMo#4978's review fixes through NVIDIA-NeMo#4905: the engine's rollout path calls the worker's _serve_rollout. The Gym path is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
left a comment
There was a problem hiding this comment.
Summary
This PR serves NeMo-Gym through the Dynamo frontend. It is stacked on #4905 (Dynamo core), which sits on #4978 (the vLLM refactor), so this review covers only this PR's own range, 1a2a581ee8..2cfb157e9a. That range also moves the Gym pin from 3f6f2059 to b0e681c: Gym#4817 plus about 260 Gym-main commits. Gym changes in that range break two CI jobs here, and both have small fixes:
- (C1) The L0 Nemo_Gym unit lane fails: 121 data-plane tests and one captured-media case still use Gym's old
rollout_idname and staging schema 2. A 3-file test fix restores the lane. - (C2) The GB200 distillation nightly fails at config load, because Gym dropped the alias that resolved
acereason-math.yaml. A one-line YAML change fixes it.
The other comments:
- (C3) Please fix in this PR: with the 1800 s affinity TTL, Dynamo's 65,536-session table fills at about 36 new captured rollouts per second, and new rollouts then get HTTP 529. No shipped config reaches that rate today. Lower the default to 300.
- (C4) The disconnect branch and the
media is Noneguard in_generate_gymhave no test; a 2-file diff adds both. - (C5)
"ng_commit_coords"is an untested copy of a Gym constant; import the constant where it is read. - (C6) The dialect test should read Gym's field names back through Gym's own config class, which drops unknown keys.
- (C7) Low: an output that vLLM ended with
finish_reason == "error"is staged before the client gets its 500; raise first, as vLLM's chat server does. - (C8) Low: the new capture test and 8 production sites still use Gym's deprecated
rollout_id; rename them in the same commit as C1. - (C9) Low: give vLLM the request's
skip_special_tokens, so that vLLM and the frontend stop at the same place. - (C10) Low: reject
extra_body.bad_wordsat setup, since the frontend answers every such request with a 400. - (C11) Doc: template kwargs sent through
extra_bodyare dropped under Dynamo. - (C12) Doc: the frontend refuses more than 4 stop strings or stop token IDs with a 500.
- (C13) Gym sends media inline; raise the frontend's 45 MB body limit and the 32 MiB request-plane limit.
- (C14) Doc:
request_timeout_s: 7200in the swe1 recipe bounds nothing now, and the exemplar comment needs the same Gym note asconfig.py. - (C15) Nit: the
_apply_dynamo_dialectdocstring says it does not mutate its argument, but it does.
Notes with no inline comment
- Follow-up to carry over. #4905's review asked to move the vLLM worker's token-capture lifecycle into its own object (
VllmRequestCapture). That code now lives in this PR: the Dynamo engine (NativeVllmEngine) calls 9 private members of the vLLM worker, 8 of them added here. Please list it under this PR's Follow-ups. - Splice coverage. #4905's C14 overlay reaches the engine_data token arrays but not the ng_prefix splice (0 of 80 rollouts made a second call;
turns_per_samplecounts one per model call). A failed splice raises an AssertionError, which becomes a 5xx that is retried and absorbed by the rollout-failure budget, and nothing asserts on it. So a frontend or template-engine bump can break the splice without turning L1 red. (The EOS-count derivation itself matches native; see below.) - swe1 nightly. The re-enabled swe1 nightly (nightly.txt#L278) runs the legacy
run_grpo_nemo_gym.py(.sh#L32). It is the first multi-engine Gym + Dynamo run, and no validation row covers it. No run is requested. - Nit. The comments at run_grpo_nemo_gym.py#L214 and run_distillation_nemo_gym.py#L114-L115 still name
dp_openai_server_base_urls; Gym now getsclient_base_urls.
What holds up
- The split from #4905 lost nothing: the head tree equals the pre-split
89469caa2dexcept for the Gym pin and 2uv.locklines. - All 27 review threads on the stack still held at
3f45420d1f, except G3, which is deferred and listed in this PR's Follow-ups. - The EOS-count derivation for the echoed prefix matches native on real Qwen3 and Nemotron-3-Nano templates.
- The disconnect branch labels its abort
client_disconnect, which is right.
Generated by Claude Code
…ing with token capture) into jthomson04/dynamo-gym Conflicts: - nemo_rl/models/generation/vllm/vllm_worker_async.py: imports keep both sides (main's VllmConfig and MultiAPIServerPool, this branch's VllmProgressReader; the chat-reply helpers stay in chat_server.py). The engine is built as on main, through the HTTP frontend pool when http_api_server_count > 1, after this branch sends the engine's chat settings to the Dynamo frontend. - tests/unit/test_recipes_and_test_suites.py: the nightly GPU-hour cap is 5392. The merged list totals 5392: main's 5296 plus 96 for the swe1 Dynamo nightly. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
vllm_cfg.http_api_server_count (NVIDIA-NeMo#5074) adds HTTP frontend processes to each engine's native server. With policy.generation.dynamo=true NeMo-Gym's calls go through the Dynamo frontend, so those processes would serve none of them. Startup now rejects the combination and points at dynamo_cfg.frontend_args.render_workers. The native path is unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
A run_test line without `fast` is skipped in every CI lane that reuses a pre-built image, so CI would not show whether this test passes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… prefix recovery) into jthomson04/dynamo-1.7 Three files conflicted: - uv.lock: re-resolved with uv 0.11.28, starting from NVIDIA-NeMo#4979's lock. Against that lock it moves ai-dynamo and ai-dynamo-runtime from 1.5.1 to 1.7.0.dev20261106, adds redis 8.1.0, and drops pydantic 2.13.0 and pydantic-core 2.46.0. Only ai-dynamo-runtime 1.5.1 pinned those two, so the lock no longer forks on them; most of the changed lines are that fork's markers going away. - docs/guides/dynamo-generation.md: this branch's wording for several frontends, with its sentences unwrapped as main's docs now are. - tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh: NVIDIA-NeMo#4979's side, which sources main's shared phase helper. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
NVIDIA-NeMo#4979 stops the Dynamo services that phase one's kill orphans from stop_phase1, which main has since moved into _gym_prefix_recovery_phases.sh. On this branch the services stop when their starter is killed (d044de8), so nothing is left to stop: the helper and the test are main's again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… engine and dynamic sampling under token capture) into jthomson04/dynamo-1.7 No conflicts. uv.lock is unchanged: the merged commits change no dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…d multimodal prefix recovery) into jthomson04/gym-actor-concurrency No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test 027b75a |
terrykong
left a comment
There was a problem hiding this comment.
Approving. The remaining validation asks are accepted without their runs: the swe1 Dynamo nightly (C22), RLVR at very high concurrency (C43 run 4) and the frontend numbers (C73). See the replies on those threads. Please watch the swe1 Dynamo nightly on main after merge.
… streaming samplers, checkpointing configs as BaseModel) into jthomson04/dynamo-gym Conflicts: - tests/unit/test_recipes_and_test_suites.py: the nightly GPU-hour cap is 5480. The merged list totals 5480: main's 5384 plus 96 for the swe1 Dynamo nightly. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
1 similar comment
…namic sampling for the streaming samplers and checkpointing configs as BaseModel) into jthomson04/dynamo-1.7 No conflicts. uv.lock is unchanged: the merged commits change no dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… two-phase SimpleStorage checkpoint capture) into jthomson04/dynamo-gym No conflicts. main now has NVIDIA-NeMo#5195, so this branch's diff against main no longer includes it, and it brings NVIDIA-NeMo#5195's code_gen fix for the Gym rollout test. The Gym pin stays 9e2e5d0ad0, which contains main's 23557b7c2a. uv.lock and pyproject.toml are unchanged and equal main's. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
1 similar comment
|
/ok to test e3aec3d |
… two-phase SimpleStorage checkpoint capture) into jthomson04/dynamo-1.7 No conflicts. uv.lock is unchanged: the merged commits change no dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
What
With
policy.generation.dynamo: true, NeMo-Gym's policy model calls the Dynamo frontend instead of each vLLM shard's HTTP server. Gym's chat calls take the native server's steps, run by the native server's code, token capture included.Based on
main: #4905 merged asf488a0c6eb, and this branch has mergedmainsince.How a Gym call is served
Dynamo's frontend runs NeMo RL's own chat engine (Dynamo's
chat_engine_factoryhook) instead of Dynamo's Rust renderer and parsers./v1/completions, which NeMo RL's own rollouts use, is unchanged.dynamo/vllm/render_pool.py): spawned processes build vLLM's renderer and chat serving from the engine args and thepolicy.generationconfig each engine publishes when it registers, parser plugins and NeMo RL's video loader included. A worker renders the call and decides how a token-capture call's prefix is served. A worker that dies (an OOM kill, say) fails only the calls it held; the pool restarts.dynamo/vllm/chat_engine.py): checks on-policy sampling, then prepares the prompt (prepare_prompt): it reads a captured prefix from TQ, splices it in at the model's turn ends and clamps the output budget. The final token IDs go through the KV router, with the prepared prompt inextra_args. A measurement call (ng_measure) gets the prompt length back without reaching an engine.DynamoVllmAdapter):start_capture(a restore, the media snapshot,begin_call), generation with theSamplingParamsvLLM's chat serving builds, sampler top-k for score centering, routes, and one TQ write (save_capture). A call the engine refuses comes back with the status the native server answers it with.nvext.nemo_rl.The steps after the render are one set of functions that both paths call (
vllm/prompt_and_capture.py):prepare_prompt,start_capture,save_capture, and one error map (error_reply) for the status, the error body, the failed-capture reason and Gym's acknowledgement. The native server runs them in the worker; behind Dynamo the frontend prepares the prompt and the engine runs the rest. They come from Terry Kong's sketch for review comment C88 (jthomson04#13), which is the first of this round's commits.frontend_args.render_workers(default 8) sets the pool size. One worker renders about 640 single-turn calls/s, 45 for a 21K-token agentic history (22 with its prefix splice), or 200 with 1–4 images (one Grace core on GB200).When the engines expose chat (
vllm_cfg.expose_http_server, which NeMo-Gym requires and the native worker starts its server on), the workers are built while the engines load (e1c510f8dc). The frontend starts them at launch, and each imports vLLM. An engine sends its settings as soon as it has its engine args, before it builds the engine, and the workers build from them. At registration the chat engine keeps the workers if the engines register the samepolicy.generation, and replaces them otherwise. On the local Gym capture test the workers are ready 10 to 12 s before the engine registers, the first chat call does not wait for them, and step 1 takes 8.0 s through Dynamo against 7.4 s native. The CMH runs under Validation predate this: there the workers started at registration, and the first step of a Gym config was 13 to 57 s slower than native. A run without the flag starts none.Rollout recovery
rollout_recovery.target_level: turnandprefix(#4506) work through Dynamo.With
prefix, a checkpoint cuts each running call and a restore resumes it from the tokens the cut saved. The engines hold the calls, so both halves run there, with the native server's code:dynamo/vllm/generation_cut.py) and sends the cut to every engine over the request plane. Each engine answers with the calls it cut, and the answers are merged into the one receipt Gym expects. The frontend sends its deadline with the cut, so an engine stops cutting in time for its answer to count. [FEATURE]: Frontend routes with POST, a request body and async handlers ai-dynamo/dynamo#18486 asks for POST routes, which would remove the port.begin_call: changed pixels are refused withretained_media_changed, as natively.The resolve is in the engine because it needs the engine's weight version and capture state. The cost is that the KV router scores a resumed call on the prompt without its saved tokens.
Changes
Shared with the native server (vLLM only; Megatron is a follow-up):
vllm/chat_server.py, unchanged.resolve_capture_input(the capture decision for a Gym candidate),on_policy_sampling,turn_end_token_idsandclamp_max_tokensare each one function both paths call.splice_prefix_tokensandreplace_prefix_tokensnow require the turn-end set.vllm/prompt_and_capture.py, above) and the reply fixes for a resumed call (ResumedCallReplyMixin) are shared. feat(checkpoint): recover active vLLM generation prefixes #4506's generation-cut route takes any backend with a bearer token and arun_cut.main. Only the Dynamo path refusesstream: trueandnother than 1, with a 400 and before anything runs; NeMo-Gym sends neither.logprobswithouttop_logprobs(omitted or null) is served as 0 on both paths, as vLLM 0.33 serves it, andtest_vllm_http_logprobs_contractis unchanged frommain.reasoning_endedis read from the template's render, not the spliced prompt, so a multi-turn call with structured output and a reasoning parser can reason before its grammar applies.max_tokens, is refused with the "maximum context length" message, which Gym markscontext_length_exceeded, instead of vLLM's "max_tokens must be at least 1". The same calls are refused as before.Gym contract:
GymRequestFields,GymCallExtrasandGymCallResult(models/generation/gym_call.py) define the fields NeMo RL and Gym exchange; the native request classes inheritGymRequestFields. Gym's Dynamo dialect sends the extras in onenemo_rl:gym:annotation and reads the result fromnvext.nemo_rl. Through Dynamo, a context-length refusal acknowledges the capture call at the end of the error message, and Gym moves it back tox-ng-capture-refused.Token capture uses Gym's
dynamo_workerbackend. Forest training and score centering work through Dynamo.frontend_args.session_affinity_ttl_secs(default 300) keeps a capture session on one engine from Dynamo 1.7 (#5115); Dynamo 1.5.1 drops a chat call's session id before the router reads it.Config:
dynamo_cfg.worker_argsstays removed, as in feat(dynamo)!: serve NeMo RL's own vLLM rollouts through Dynamo's backend SDK #4905, and is rejected at startup: Gym chat uses the nativevllm_cfgchat settings.frontend_args.render_workers.extra_cli_argsmay not set--dyn-chat-processoror--custom-jinja-template.DYN_IGNORE_OPENAI_FE_UNSUPPORTED_FIELDS=0is pinned, so the frontend never drops an unknown field silently.Also rejected at startup with
dynamo: true:async_rl.generation_router.enabled: the frontend already routes the fleet.bad_wordsinextra_bodyorsampling_overrideson the policy model: the frontend answers it with a 400.stop_token_ids,allowed_token_ids,bad_words_token_ids,cache_salt,logprob_token_idsordetokenizein either block on the policy model: the frontend accepts them but never hands them to a Python chat engine.vllm_cfg.skip_tokenizer_init: truewith NeMo-Gym: the engine detokenizes Gym's replies.vllm_cfg.http_api_server_countabove 1 (feat(vllm): share one engine across multiple HTTP frontends #5074's extra vLLM HTTP frontends per engine): Gym's calls go through the Dynamo frontend, so those processes would serve none of them.frontend_args.render_workersis the Dynamo-side setting.Differences from native, also in the guide:
structured_outputs,priority, ...), with a 400.CI:
L1_Functional_Tests_Dynamo.shruns every overlay in fast mode: math, Gym with and without capture, reload refit, context compaction (forest training), router replay with deferred routes, turn recovery, and prefix recovery (C94).Merge order
mainas8f3595a112, and merged into this branch ate3aec3d1ce, with itscode_genfix.440104210.mainat9e2e5d0ad: the Gym#4817 merge plus five later commits.mainasbb9ed784e4, and merged into this branch at7e4fbd5ea4. This PR lands second, sostart_capturenow makes feat(checkpoint): recover multimodal generation prefixes #4583's restored-media call (C92), which both paths run.Against the previous pin (
6dc68d2fa, on Gym#4817's branch), the new one adds 39 other Gymmaincommits. Thevllm_modelserver is the same at the merge commit, and Gym's dependencies are the same, souv.lockdoes not change. The pin contains the commit #5195 pins (23557b7c2).Since the functional and cluster runs under Validation, Gym#4817's review added four commits to the dialect:
reasoning_field: bothis refused withdynamo_frontendonly when the Gym server replays reasoning. NeMo RL's setup setsreasoning_content, so its policy model never reached the check.logprob_token_idskeeps Gym's HTTP 400 through Dynamo. This PR's startup check covers the same field only in NeMo RL's own policy-model overlay (extra_bodyandsampling_overrides).forward_session_id_as_conversation_idis refused withdynamo_frontend, because the frontend rejects theconversation_paramsfield it sends. NeMo RL reads that field only in its TensorRT-LLM server./tokenizecount. This PR's chat engine lowers the budget itself, so the count was never used.None of these changes a call this PR's recipes make.
On the re-pin (
766f81d004), run locally:uv lock --checkpasses.test_l0_gym_environments_roll_out_through_nemo_rl. They fail at setup because of the bump, not this PR: since refactor(code-gen): replace legacy Ray execution with sandboxes Gym#4958, Gym'scode_genserver requiressandbox_config.image. build(gym): bump the Gym submodule #5195 now has a commit for that (5ff3ddfd25).test_recipes_and_test_suites.pyandtest_config_validation.py: 451 passed.The other functional and cluster runs under Validation used an earlier pin (
9aac609, the same dialect commits on the older Gym base) and were not repeated, except the multimodal comparison on vLLM 0.33, which ran on the current pin.Validation
Step-order test (
tests/unit/models/generation/test_gym_chat_path.py, 36 cases). Each sends one call down both paths: the native routes over ASGI, and our chat engine, a spawned render pool and the adapter. It checks they run the same prompt, media andSamplingParams, stage the same capture record, and return the same reply. The cases cover:nano_v3's parsers, every sampling field, and refusals;max_tokens, a tool schema with a null default, and a namedtool_choiceunder a parser without a structural tag;top_logprobsomitted or null, served as 0 on both paths;reasoning_endedafter the model's own earlier turn, and router replay's staged routes.Each case fails when the code it covers is reverted.
Functional runs on 2× RTX PRO 6000. Each native SingleController test passed with
++policy.generation.dynamo=trueandcheck_dynamo_overlay.sh. The first three rows ran at45add8b0a1, the rest at817107f3a0:grpo_dp_single_controller.shgen_kl_error0.0006grpo_async_gym_single_controller.shwith capturegen_kl_error0.0011grpo_async_gym_single_controller_context_compaction.sh(forest training)gen_kl_error≤ 0.0011. At817107f3a0: of 128 chat calls, 20 candidates served token-in and 24 rooted new segmentsgrpo_async_gym_single_controller.shwithout capturegrpo_dp_single_controller_reload_refit.shgen_kl_error0.0003nemotron_omni_clevr_vllm_token_capture_single_controller_1n2g.sh(VLM, media capture)media_row_rate1,gen_kl_error0.0004. The overlay check's KV-event line fails only because this config turns prefix caching offPrefix recovery (at
26d9d01bbb):test_gym_chat_path.pycuts a call with the worker's own cut, resumes it natively and through Dynamo, and compares both with the uninterrupted call: the prompt andSamplingParamsthe engine ran, the staged row, the reply's content, reasoning and tool calls, and usage. The cut lands inside the reasoning, after it, inside the tool call, and after the call finished; one case loses the cut's rows and regenerates.test_dynamo_render_pool.py, 8 cases × 2 tokenizer styles).test_dynamo_generation_cut.pyfans a cut out to four engines (one holds a call, one holds none, one raises, one answers after the deadline). Against Dynamo 1.5.1 with the real frontend launcher and three stub engines, one of them slow: 404 before capture is set up, 401 for a wrong bearer, every engine sent the same deadline, an answer at that deadline, and a receipt that passes Gym'svalidate_forwith only the slow engine's call failed.8ca22f074e, not part of this PR) passes natively (504 s) and with++policy.generation.dynamo=true(517 s): all 8 rollouts resumed, log probabilities bit-identical to the uninterrupted run, andcheck_dynamo_overlay.sh. Turn recovery behind Dynamo passes (265 s).766f81d004, on Gymmainand with the test now in the repo: it passes natively (1,291 s; the first run, which also rebuilt Gym's server venvs) and with++policy.generation.dynamo=true(804 s), with the same checks. Behind Dynamo,stop_phase1also stops the services that phase one's kill orphans, so the overlay's leak check passes.Rerun at
26d9d01bbb, after the shared steps landed:grpo_async_gym_single_controller.shwith capture passes natively (114 s) and through Dynamo (117 s; 251 KV events applied,gen_kl_error0.0011), andgrpo_dp_single_controller.shthrough Dynamo (64 s; 393 KV events applied).Unit tests at
26d9d01bbb, run without-xso one failure cannot hide the rest: vLLM lane 119 (the same 119 withnemo_gymnot importable, as in CI's vLLM lane), vLLM+Gym lane 66 and Gym lane 1,203 passed. The default lane (4,371 passed) fails only locally: nonemo_automodel, and the Ray actors that leaves behind. CI's pre-commit hooks, run over every file this PR changes, and project-mode pyrefly are clean.Merge with
main(e6d892c552, which bumps vLLM from 0.29.0 to 0.33.0; the commit before #4583), ate449a14141. Run locally on a venv synced from the merged lock (vLLM 0.33.0, torch 2.14.0, ai-dynamo 1.5.1):code_generrors, config tests 548 passed.test_vllm_generation.py: 196 passed. The other 10 neednemo_automodel, which the vLLM venv cannot hold.grpo_async_gym_single_controller.shwith capture through Dynamo on 2 GPUs passes, withcheck_dynamo_overlay.sh(gen_kl_error0.0011, 224 KV events applied).What the merge needed beyond its conflicts:
main's two vLLM 0.33 changes to the chat handler are applied inchat_server.py, where this PR moves that code, and the nightly GPU-hour cap goes from 5310 to 5318, becausemainadded an 8-hour nightly under the cap this PR's swe1 nightly had filled.Merge with
mainagain (bb9ed784e4, #4583), at7e4fbd5ea4, on the same venv:code_generrors, config tests 548 passed.test_vllm_generation.pyas above.start_capture. Two new cases cover the Dynamo engine: a restored image call stages its reprocessed pixels, or is refused withretained_media_changedbefore it begins or generates; and a cut writes a media call's tensor digests into its first prefix row. With the restored-media call removed, feat(checkpoint): recover multimodal generation prefixes #4583's test and both restore cases fail.++policy.generation.dynamo=true(765 s): all 8 rollouts resumed, log probabilities bit-identical to the uninterrupted run, andcheck_dynamo_overlay.sh(1,662 KV events applied). feat(checkpoint): recover multimodal generation prefixes #4583 movesstop_phase1into the shared_gym_prefix_recovery_phases.sh, and the Dynamo cleanup moved there with it.Third merge with
main(705953be86: #5074, several HTTP frontends per engine, and #5202, dynamic sampling with token capture), at936f5d28c6; head027b75a1f1, on the same venv:code_generrors, config tests 550 passed.test_vllm_generation.py: 207 passed, with the same 10 that neednemo_automodel.grpo_async_gym_single_controller.shwith capture on 2 GPUs passes through Dynamo (145 s; 82 requests routed, 218 KV events applied,gen_kl_error0.0010) and natively withhttp_api_server_count=2(205 s;gen_kl_error0.0011). The second run puts feat(vllm): share one engine across multiple HTTP frontends #5074's spawned frontend on the chat server this PR moves tochat_server.py.++policy.generation.dynamo=true(589 s; 40 requests routed, 1,694 KV events applied) and natively (641 s): 8 rollouts resumed, log probabilities bit-identical over 9,312 tokens.What the merge needed beyond its conflicts: the nightly GPU-hour cap is 5392, the merged list's total (
main's 5296 plus 96 for the swe1 Dynamo nightly), andhttp_api_server_countabove 1 is rejected withdynamo: true(above).Fourth merge with
main(aa0f77af51: #5206 group length penalties, #5036, #5226, #4784 dynamic sampling for the streaming samplers, #5239 checkpointing configs as BaseModel, and #5244), at4581367100:mainchanged no generation code,pyproject.toml,uv.lockor submodule pin, so only the tests the merge could affect were rerun, on the same venv: config tests 555 passed, and the three test files changed on both sides (test_setup.py,test_grpo.py,test_distillation.py) 861 passed.The one conflict was the nightly GPU-hour cap, now 5480, the merged list's total (
main's 5384 plus 96 for the swe1 Dynamo nightly).Fifth merge with
main(8f3595a112: #5137 two-phase SimpleStorage checkpoint capture, and #5195 the Gym bump), ate3aec3d1ce:9e2e5d0ad0, which contains the pin build(gym): bump the Gym submodule #5195 brought tomain.test_rollouts.py107 passed, and the Gym-marked run of the four Gym test files 85 passed, including the 14code_genrollout cases that failed before this merge.Native vs Dynamo validation (CMH, GB300)
Each config ran three ways with identical overrides: (a)
mainat27fac5f3d6, native vLLM; (b) this stack with Dynamo off, at #51173bab36fee4; (c) this stack withpolicy.generation.dynamo=true, at #51178a5a90ef7a, which sits on this PR's56774f269d. #5117 carries this PR and #5115, so both stack arms are on ai-dynamo1.7.0.dev20261106, not the 1.5.1 this PR pins. The two stack commits differ only in the Dynamo frontend's chat workers. Runs: nvidia/nemo-rl-dynamo-parity.The test scripts run unchanged, with more steps. The score-centering and Nano tests cannot train as written (one read-only tool, Qwen3-0.6B, the Nano Base checkpoint), so those three use the instruction-tuned model each config names, all 27 workplace-assistant tools, a 16,384-token context and 8 prompts × 16 rollouts per step, in every arm.
Forest training (context compaction), Qwen3-0.6B (C43 run 1;
forest)gen_kl_error, mediantoken_mult_prob_error, medianScore centering, Qwen3-30B-A3B-Instruct-2507 (C43 run 2;
score-centering-instruct)gen_kl_error, mediantoken_mult_prob_error, medianScore centering with filtered sampling and sampler replay, Qwen3-30B-A3B-Instruct-2507 (C43 run 2;
score-centering-filtered-instruct)gen_kl_error, mediantoken_mult_prob_error, medianMultimodal (Omni CLEVR, media token capture) (C43 run 3;
omni-vlm)gen_kl_error, mediantoken_mult_prob_error, medianMultimodal, repeated on vLLM 0.33 (
omni-vlm, at027b75a1f1)The table above ran on vLLM 0.29. This repeats it after the merges with
main, with the same test, overrides, cluster and node shape: (a)mainat705953be86, themaincommit this PR contained at027b75a1f1; (b) this PR at027b75a1f1with Dynamo off; (c) the same commit withpolicy.generation.dynamo=true. All three are on vLLM 0.33.0. Arm (c) is on ai-dynamo 1.5.1, the version this PR pins, where the other tables in this section used the 1.7 nightly.gen_kl_error, mediantoken_mult_prob_error, medianAll three runs train 16 steps and pass every gate the test defines. In arm (c) the frontend routed all 256 requests, and no frontend, etcd or NATS process was left on the node.
gen_kl_error: the Dynamo run's median was the highest of the three on vLLM 0.29 and is the lowest here.e1c510f8dcthe frontend's chat workers are built while the engines load, and the first chat call waited 0.0 s for them.policy_training8.1 s against 6.9 and 7.6 s), not in generation or weight sync, and the order is the reverse in the table above. With one run per arm it cannot be told apart from node-to-node spread.main(bb9ed784e4), whoseuv.lockis identical to705953be86's. This PR pins a newer Gym that requiresopenai2.45.0, so arms (b) and (c) put thatopenaionPYTHONPATHand build their own Gym server venvs. An image built from this PR's lock would need neither.027b75a1f1also passes on 2 GPUs, natively and through Dynamo, with vLLM given 85% of a GPU (the model does not fit in the YAML's 60% of a 96 GB card).Nano 30B instruct with Gym: tool and reasoning parsers, token capture, deferred R3 (C22;
nano-gym-instruct)gen_kl_error, mediantoken_mult_prob_error, medianR3 on text rollouts (PPO, Qwen3-30B-A3B) (extra;
r3-ppo)gen_kl_error, mediantoken_mult_prob_error, medianEvery gate each test defines passes in all three arms, with two exceptions that are the same in every arm: gates that count steps (these runs are longer), and the filtered score-centering test's
max(token_mult_prob_error) < 1.05(see the third note). The Dynamo arm's routing check (check_dynamo_overlay.sh) confirms requests went through the frontend in every config.Four notes:
56774f269dit was 60 to 130 s. On this cluster the render pool's first worker takes 67 to 86 s to start, longer than the time between engine registration and the first chat call. Later steps match, and text rollouts show no delay.e1c510f8dcbuilds the workers while the engines load instead. Only the multimodal config has been rerun with it on this cluster (the vLLM 0.33 table above), and there the Dynamo run's first step is no slower than native.{"message":"Bad Request"}in place of the chat engine's message and capture acknowledgement, so Gym fails the call and the row is redispatched (2 and 5 rows above).token_mult_prob_errorjumps on a few steps in every arm,mainincluded, while the median stays near 1.02. Steps above 1.1, of 40 per arm: 5 onmain, 5 with Dynamo off, 6 with Dynamo on. The Nano config, which replays routed experts, has none.Not covered here: RLVR at very high concurrency (C43 run 4), a subagent config (none exists in the repo), and the swe1 nightly itself (legacy path; the Nano config above covers what it tests).
Before merge
mainand on this PR with Dynamo off and on (C43 run 4). The other three runs are under Validation; forest training ran without subagents, since no config in the repo sets them up. Still to report from a Dynamo-on run: frontend chat calls/s, render-pool queue wait (the agentic one on more engines thanrender_workers) and the frontend's TQ prefix-read rate, which one prefix cache now serves for the whole fleet. The runs above do not log them.mainand merged into this branch, so thecode_genrollout test passes again (see Merge order). The Gym re-pin (C32) is done.Follow-ups
++policy.generation.dynamo=true. It is not in the Dynamo L1 list, and a restored image call through Dynamo has unit coverage only.reasoning_endedfor a resumed call: a call with a structural-tag tool parser that was cut after its own</think>is told its reasoning has not ended, so the tag is not applied to the resumed tokens. Reading it from the template's ids plus the call's saved tokens needs the reasoning parser on the worker behind Dynamo.session_affinity_ttl_secson by default needs measuring at scale, on Dynamo 1.7.🤖 Generated with Claude Code