diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym
index 23557b7c2a4..9e2e5d0ad0d 160000
--- a/3rdparty/Gym-workspace/Gym
+++ b/3rdparty/Gym-workspace/Gym
@@ -1 +1 @@
-Subproject commit 23557b7c2a48c60975c7719ee25cb6adb42963b7
+Subproject commit 9e2e5d0ad0d1723acef1b63beb28292da16f7c5b
diff --git a/docs/design-docs/dynamo-integration.md b/docs/design-docs/dynamo-integration.md
index b1a292a7308..87a97bb07e7 100644
--- a/docs/design-docs/dynamo-integration.md
+++ b/docs/design-docs/dynamo-integration.md
@@ -5,7 +5,12 @@
- A vLLM bump must never be blocked by Dynamo.
- Dynamo CI coverage must stay consistent with native vLLM's.
-Both hold only if the engine process contains no Dynamo-owned vLLM code. A vLLM bump can then break Dynamo serving only in NeMo RL's own vLLM adapter (`vllm_adapter.py`), which calls a few vLLM APIs native NeMo RL does not: the `get_kv_cache_group_metadata` engine utility and `KVCacheSpecKind` at registration, and `StatLoggerBase` for the router's load signal. A renamed import fails the vLLM-lane unit tests at collection, and the Dynamo functional overlays run registration.
+Both hold only if no process runs Dynamo-owned vLLM code. A vLLM bump can then break Dynamo serving only in NeMo RL's own `dynamo/vllm/` package:
+
+- the engine adapter (`adapter.py`) calls a few vLLM APIs native NeMo RL does not: the `get_kv_cache_group_metadata` engine utility and `KVCacheSpecKind` at registration, and `StatLoggerBase` for the router's load signal;
+- the frontend's chat engine (`chat_engine.py`, `render_pool.py`) builds the same vLLM renderer and chat serving the native server builds (`vllm/chat_server.py`), from the engine's own engine args.
+
+A renamed import fails the vLLM-lane unit tests at collection, and the Dynamo functional overlays run registration and NeMo-Gym's chat calls.
## Engine hosting
@@ -13,15 +18,20 @@ Both hold only if the engine process contains no Dynamo-owned vLLM code. A vLLM
`dynamo.common.backend` is Dynamo's engine-agnostic backend SDK and imports no vLLM. NeMo RL implements its `LLMEngine` in two parts:
-- `dynamo/engine_adapter.py`: `DynamoEngineAdapter`, the Dynamo side shared by every generation backend. It serves request kinds as handlers keyed by their annotation, maps engine errors to a 400 that keeps the cause, and registers the endpoint for `/v1/completions` only. It imports no backend, and the engine only ever receives token IDs.
-- `dynamo/vllm_adapter.py`: `DynamoVllmAdapter`, the vLLM backend over the worker's `AsyncLLM`. A backend implements `serve_rollout`, `start`, `abort` and `kv_event_sources`.
+- `dynamo/engine_adapter.py`: `DynamoEngineAdapter`, the Dynamo side shared by every generation backend. It serves request kinds as handlers keyed by their annotation, maps engine errors to a 400 that keeps the cause, and registers the endpoint for `/v1/completions` (NeMo RL's rollouts) and chat (NeMo-Gym). It imports no backend, and the engine only ever receives token IDs.
+- `dynamo/vllm/adapter.py`: `DynamoVllmAdapter`, the vLLM backend over the worker's `AsyncLLM`. A backend implements `serve_rollout`, `start`, `abort` and `kv_event_sources`, and may add request handlers.
The two together:
-- `generate` serves only the rollouts NeMo RL itself sends, which carry an annotation, and refuses other requests. The worker's own `_serve_rollout` runs each rollout exactly as native `generate_async` does, and `request_pipeline.rollout_result` builds its result, routed experts included.
+- `generate` maps Dynamo's token request onto vLLM `SamplingParams` the way vLLM's OpenAI server does, and streams token deltas back. Rollouts that NeMo RL itself sends carry an annotation. For those, the engine uses the same code as native `generate_async`:
+ - the worker's `_rollout_max_new_tokens` and `_rollout_sampling_params` build the request;
+ - `request_pipeline.rollout_result` builds the result, routed experts included.
+
The result travels in `engine_data`, and the driver assembles the same output with `request_pipeline.rollout_output`.
+
+ NeMo-Gym's calls arrive rendered and spliced by NeMo RL's chat engine (see [NeMo-Gym chat](#nemo-gym-chat)), annotated for the adapter's Gym handler, with the capture plan in `extra_args`. The handler runs the native server's steps after the render, with the worker's own helpers: the media snapshot (`_capture_request_media`), `begin_call` (which stamps the weight version), the generation with the `SamplingParams` vLLM's chat serving builds from the request, and the one TQ write. Its `engine_data` carries the output the frontend builds the reply from, and the call's `GymCallResult`. A call the engine refuses before it runs (a vLLM request error, a full queue, media capture refused) comes back in `engine_data` too, with the status the native server answers it with: Dynamo hands a raised error to the chat engine without its status.
- `abort` cancels the vLLM request on client disconnect.
-- Registration advertises the context length, the KV block size (from vLLM's KV-cache group metadata), the block count and the scheduler limits. It publishes no token budget. The engine owns the context check and the output cap, as the native server does. With a budget, the frontend would refuse NeMo RL's own rollouts that fill the context.
+- Registration advertises the context length, the KV block size (from vLLM's KV-cache group metadata), the block count and the scheduler limits. It publishes no token budget. The engine owns the context check and the output cap, as the native server does. With a budget, the frontend would fill an omitted `max_tokens` from the prompt it rendered, which a prefix splice then changes, and would refuse NeMo RL's own rollouts that fill the context.
- `kv_event_sources` points Dynamo at the engine's own vLLM KV-event ZMQ publisher. It is set only when the engine caches prefixes. Each engine binds its own Linux abstract-namespace socket (`ipc://@...`), so engines on one node need no port and cannot collide. The name is random rather than the pid, because containers that share the host network share these names. vLLM refuses `ipc://` only above data-parallel rank 0, and vLLM data parallelism is rejected at startup with Dynamo. A vLLM stat logger publishes the router's per-engine load signal.
Because the Worker runs on the actor's own loop, the worker's other methods interleave with Dynamo requests exactly as with native requests:
@@ -32,6 +42,38 @@ Colocated generation is rejected at startup: a sleeping engine stays registered,
No HTTP control routes are involved. The Rust Worker installs a SIGTERM handler that chains to Ray's, so actor shutdown is unchanged, and discovery drops an engine when its process exits.
+## NeMo-Gym chat
+
+Dynamo's frontend hands chat calls to a Python `chat_engine_factory` when one is given. `dynamo/vllm/frontend.py` starts `dynamo.frontend`'s own `main` with NeMo RL's factory added to the `EntrypointArgs` it builds, so the frontend keeps its HTTP server, router and `/v1/completions` path, and only chat goes to NeMo RL's chat engine (`dynamo/vllm/chat_engine.py`).
+
+Each engine publishes, in its registration's `runtime_data`, its own engine args and the `policy.generation` chat settings. The chat engine builds a render pool (`dynamo/vllm/render_pool.py`) from them: `render_workers` spawned processes, each holding the vLLM renderer and chat serving the native server builds. Rendering is CPU-bound Python, and the frontend serves every call on one event loop, so it runs in the pool, first in first out.
+
+When the engines expose chat (`vllm_cfg.expose_http_server`, which NeMo-Gym requires and the native worker starts its server on), the workers are built while the engines load, so NeMo-Gym's first calls do not wait for them:
+
+- the frontend starts the workers at launch, and each imports vLLM;
+- an engine sends the driver its settings as soon as it has its engine args, before it builds the engine. The driver hands them to the frontend in a file, as it does the capture config, and the workers build from them;
+- at registration the chat engine keeps the workers if the engines register the same `policy.generation`, and replaces them otherwise. It does not compare the engine args, which vLLM adds to while it builds the engine (scheduler limits, resolved model paths; no render uses those). Workers that got no settings before registration build then.
+
+A run without chat starts none. Per call:
+
+1. the pool renders it with the native server's render step (`_render_chat_call`, shared with `vllm/chat_server.py`), including the token-capture decision for a Gym candidate, the history render the splice cuts against, and `reasoning_ended`, read from the template's render and never from a spliced prompt;
+2. the chat engine checks on-policy sampling (`on_policy_sampling`, shared with the native route), reads a token-in prefix from TQ through a read-only client, splices it in with the model's turn-end tokens, and clamps the output budget (`prepare_prompt` in `vllm/prompt_and_capture.py`, shared with the native server);
+3. the final token IDs go through the KV router to one engine, whose adapter runs the native server's `start_capture` and `save_capture`;
+4. the pool builds the reply with vLLM's own response builder and parsers, through the same reply fixes for restored calls as the native server.
+
+A call that is refused or fails is answered through one map on both paths (`error_reply` in `vllm/prompt_and_capture.py`): the status, the error body, the reason its capture is failed with, and whether Gym is told the capture call was refused before it was staged.
+
+`GymCallExtras` and `GymCallResult` (`models/generation/gym_call.py`) define the fields both paths exchange with Gym. `tests/unit/models/generation/test_gym_chat_path.py` sends the same calls through the native routes and through this path and checks they match step for step.
+
+### Prefix recovery
+
+With `rollout_recovery.target_level: prefix`, a checkpoint cuts each running call (its decoded tokens are staged in TQ), and a restore resumes it from those tokens. The engines hold the calls and the worker-side state (`models/generation/prefix_recovery`), so both halves run there, with the native server's code:
+
+- **The cut.** NeMo-Gym posts it beside the URL its calls go to, the frontend. A Dynamo frontend route can only be a synchronous GET, so `dynamo/vllm/frontend.py` also wraps `DistributedRuntime` and serves the native route (`vllm/generation_cut_route.py`) on a second port, in the frontend process (`dynamo/vllm/generation_cut.py`). It sends the cut to every engine with `Client.direct`; the adapter's `nemo_rl:generation_cut` handler runs the worker's own cut and answers with the calls it cut; and the answers are merged into the one receipt Gym expects. A call no engine cut gets Gym's failure ack and regenerates after a restore. The frontend takes one deadline when the cut arrives and sends it along, so an engine stops cutting in time for its answer to reach the frontend; an engine that timed the cut itself would start later and answer too late.
+- **The resume.** The frontend renders and routes a re-sent call like any other. The adapter then runs the steps the native server runs (`start_capture` in `vllm/prompt_and_capture.py`): it reads and checks the saved rows, replaces the prompt with them, lowers the output budget, checks the call's reprocessed media against the digests the cut saved, and begins capture with the cut. A cut it cannot use (rows gone, or newer than its weights) is dropped, and the frontend's prompt runs. The engine's reply names what it resumed from, and the render pool puts the saved text back before vLLM's parsers run and counts the saved tokens as output (`prefix_recovery/restore.py`).
+
+The resolve is in the engine, not the frontend, because the check needs the engine's weight version and its capture state, and a cut the engine drops then needs no second pass through the frontend. The cost is that the KV router scores a resumed call on the prompt without its saved tokens.
+
## Driver
`VllmGeneration` starts `ManagedDynamoRuntime`, which runs etcd, NATS and `dynamo.frontend`. After the engines load, it asks every DP leader to register and waits until the frontend routes to all of them. Each poll also asks the leaders whether their endpoint task stopped, so a failed registration fails with its cause instead of a timeout.
@@ -40,12 +82,13 @@ No HTTP control routes are involved. The Rust Worker installs a SIGTERM handler
## Dependency coupling
-The vLLM workers' venv (the `vllm` extra) carries the released `ai-dynamo` and `ai-dynamo-runtime` wheels, never `ai-dynamo[vllm]`. The engine process only imports `dynamo.common.backend`, `dynamo.llm.exceptions` and `dynamo._core`.
+The vLLM workers' venv (the `vllm` extra) carries the released `ai-dynamo` and `ai-dynamo-runtime` wheels, never `ai-dynamo[vllm]`. NeMo RL imports only the Dynamo modules `tests/unit/models/generation/test_dynamo_vllm_decoupling.py` lists: the backend SDK, `dynamo.llm` and the frontend's `main`, none of which import vLLM.
What a Dynamo upgrade has to re-check:
-- the `dynamo.common.backend` contract (`LLMEngine`, `Worker`, `WorkerConfig`, publisher);
-- the request fields of Dynamo's preprocessed request;
-- the frontend's `nvext` extensions and annotation forwarding.
+- the `dynamo.common.backend` contract (`LLMEngine`, `Worker`, `WorkerConfig`, publisher and logprob helpers);
+- the request fields of Dynamo's preprocessed request, and `extra_args` reaching the engine;
+- the frontend's `chat_engine_factory` hook and the `EntrypointArgs` name `dynamo/vllm/frontend.py` wraps, which are internal API;
+- the Python chat request the hook receives, its `nvext` and annotation fields, and `EngineConfig.runtime_data` reaching the frontend.
The one vLLM↔Dynamo wire coupling is vLLM's KV-event msgpack, which Dynamo's router decodes. A change there degrades routing, not serving. The managed runtime logs the router's applied-event counts at shutdown, and the Dynamo functional overlays fail if the engines published events but none were applied (`tests/functional/check_dynamo_overlay.sh`).
diff --git a/docs/guides/dynamo-generation.md b/docs/guides/dynamo-generation.md
index 627192b7901..2f454ec4a01 100644
--- a/docs/guides/dynamo-generation.md
+++ b/docs/guides/dynamo-generation.md
@@ -21,6 +21,7 @@ With `dynamo: true`:
- `vllm_cfg.async_engine` is turned on, because Dynamo drives the engine's `AsyncLLM`. Settings Dynamo does not serve are rejected at startup; see [Not supported](#not-supported).
- The driver starts etcd, NATS and `dynamo.frontend`. The frontend runs from the vLLM workers' own venv, so it is the same ai-dynamo release.
- Each engine turns on vLLM's KV-cache events for `router_mode: kv` and registers with the frontend from inside its own worker process. Setup waits until the frontend routes to every engine.
+- With NeMo-Gym (`vllm_cfg.expose_http_server: true`), the frontend starts its render workers at launch and builds them while the engines load, from the settings one engine sends before it loads. Gym's first calls then do not wait for them. Other runs start none.
- vLLM publishes KV-cache events only for blocks its prefix cache stores. With `vllm_cfg.enable_prefix_caching: false`, the engines publish none, and the KV router routes on load alone.
- Shutdown stops the services after the engines.
@@ -33,7 +34,9 @@ With `dynamo: true`:
| `startup_timeout_s` | 600 | Bounds service start-up and engine registration |
| `request_timeout_s` | null | Deadline for one NeMo RL rollout through the frontend. null waits as long as generation takes, as native does; on the SingleController, `async_rl.rollout_failure.native.generation_timeout_s` bounds both paths |
| `frontend_args.router_mode` | `kv` | Frontend routing policy |
-| `frontend_args.extra_cli_args` | `[]` | Extra `dynamo.frontend` flags |
+| `frontend_args.session_affinity_ttl_secs` | 300 | Keeps a NeMo-Gym token-capture rollout on one engine while it is idle for up to this many seconds; null disables it. Takes effect from Dynamo 1.7: Dynamo 1.5.1 drops the session id of a chat call served by NeMo RL's chat engine. Dynamo keeps at most 65,536 sessions and answers a new one past that with HTTP 529 |
+| `frontend_args.render_workers` | 8 | Processes in the frontend that render NeMo-Gym's chat calls and build their replies; every Gym call in the fleet shares them. Each holds its own tokenizer and processor, about 1.7–2.3 GB. On one Grace core of a GB200, one serves about 640 single-turn calls/s, 45 for a 21K-token agentic history (22 with its prefix splice), or 200 with 1–4 images |
+| `frontend_args.extra_cli_args` | `[]` | Extra `dynamo.frontend` flags. `--dyn-chat-processor` and `--custom-jinja-template` are rejected: NeMo RL renders chat itself |
The etcd and nats-server binaries are found in `NEMO_RL_DYNAMO_BIN_DIR` when it is set (the image sets it to `/opt/dynamo/bin`), and on `PATH` otherwise.
@@ -44,21 +47,52 @@ The etcd and nats-server binaries are found in `NEMO_RL_DYNAMO_BIN_DIR` when it
| `generate_async` on text token prompts, router replay (R3) included | Frontend (KV-aware); the engine samples and assembles it exactly as native `generate_async` does |
| NeMo RL's own multimodal rollouts | Native shard dispatch. Those rollouts send media to the engine directly rather than as OpenAI content |
| `generate_text_async`, which evaluation uses | Native shard dispatch |
-| NeMo-Gym's OpenAI requests | Not served yet; see [Not supported](#not-supported) |
+| NeMo-Gym's chat calls | Frontend, rendered by NeMo RL's chat engine with the native server's code. See [NeMo-Gym](#nemo-gym) |
A rollout through the frontend has the same `GenerationOutputSpec` as the native path: token IDs, log probabilities, truncation, routed experts with their `r3_routed_experts_*` counts, and `gen_leader_worker_idx`. Both paths build it with `nemo_rl/models/generation/vllm/request_pipeline.py`.
-The engines serve only these rollouts; the frontend refuses other requests.
+## NeMo-Gym
+
+NeMo-Gym's policy model calls the frontend. Setup turns on Gym's Dynamo dialect (`dynamo_frontend` in Gym's [`vllm_model`](https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_models/vllm_model) config). The frontend takes only OpenAI fields plus `nvext`, so Gym sends the native request's extra fields (`ng_capture`, `required_prefix_token_ids`), and `ng_measure` for the measure call it makes in place of `/tokenize`, in one `nemo_rl:gym:` annotation, and reads the call's token fields (`prompt_token_ids`, `generation_token_ids`, `generation_log_probs`, `routed_experts`, `ng_commit_coords`) from `nvext.nemo_rl`.
+
+The frontend serves chat with NeMo RL's chat engine instead of Dynamo's own renderer and parsers (Dynamo's `chat_engine_factory` hook). Each call takes the native server's steps, with the native server's code:
+
+1. In the frontend, a render-pool process renders the call with the engine's own tokenizer, template, processor and `vllm_cfg` chat settings (parser plugins included), which each engine publishes when it registers. It also decides how a token-capture call's prefix is served.
+2. The chat engine checks on-policy sampling, reads a captured prefix from TQ, splices the prefix into the render, clamps the output budget to the context, and routes the final token IDs through the KV router.
+3. On the engine, the adapter snapshots the call's media, begins its capture, generates with the `SamplingParams` the native server builds, and stages the call in one TQ write. Only the commit coordinates travel back.
+4. A render-pool process builds the reply with vLLM's reasoning and tool parsers.
+
+`tests/unit/models/generation/test_gym_chat_path.py` sends the same calls down both paths and checks they run the same prompt and `SamplingParams`, stage the same record and return the same reply.
+
+Where the behavior differs from the native endpoint:
+
+- The frontend refuses more than 32 `stop` strings, and vLLM request fields its own request type lacks (for example `structured_outputs`, `return_token_ids`, `truncate_prompt_tokens`, `priority`, `chat_template`), with HTTP 400.
+- gpt-oss (Harmony) chat calls are refused with HTTP 400.
+- A call that streams (`stream: true`) or asks for more than one choice (`n`) is refused with HTTP 400. The native endpoint serves both, and fails the capture of a streamed token-capture call. NeMo-Gym sends neither.
+- A token-capture call refused for its context length acknowledges the refusal at the end of the error message instead of in the native endpoint's `x-ng-capture-refused` header, which Dynamo's errors cannot carry. Gym's dialect moves it back to the header, so the harness sees the native error.
+- A call's processed media travels to the engine inside the request.
+
+### Rollout recovery
+
+`rollout_recovery.target_level: turn` and `prefix` work through Dynamo with no extra setting. With `prefix`, NeMo-Gym asks the generation backend to cut its running calls at each checkpoint, and re-sends each cut call after a restore.
+Two things differ from the native endpoint:
+
+- **Where the cut goes.** Gym sends the cut beside the URL its calls go to, which is the frontend, but the calls run in the engines. The frontend process takes the cut on a port of its own, from the same range as etcd and NATS, and sends it to every engine over Dynamo's request plane; each engine cuts the calls it holds, up to the frontend's deadline (`token_capture.control_timeout_s` from the cut's arrival there). Setup tells Gym the port (`generation_cut_control_url` in Gym's `vllm_model` config). The port is needed until a Dynamo frontend route can take a POST ([ai-dynamo/dynamo#18486](https://github.com/ai-dynamo/dynamo/issues/18486)).
+- **What the router sees for a resumed call.** The engine replaces the prompt the frontend sent with the call's saved tokens and generates the rest, and the frontend puts the saved text back in front of the reply. So the KV router scores that one call on a prompt without the saved tokens. A cut the engine cannot use is dropped and the call regenerates, as natively.
## Not supported
Startup rejects these with `dynamo: true`:
-- NeMo-Gym (`env.should_use_nemo_gym`): the frontend does not serve Gym's calls yet. Run Gym recipes with `dynamo: false`.
- `colocated.enabled: true`: a colocated engine sleeps while the policy trains, and the frontend would keep routing to it.
- `async_rl.generation_fleet_health.enabled`: the frontend picks engines itself, and a restarted shard does not register with it yet.
+- `async_rl.generation_router`: the frontend already routes the fleet behind one URL.
+- `bad_words` in `extra_body` or `sampling_overrides` on the policy model (`env.nemo_gym.policy_model.responses_api_models.vllm_model`): the frontend answers every request carrying it with HTTP 400. Startup checks only this overlay, not Gym's own YAMLs.
+- `stop_token_ids`, `allowed_token_ids`, `bad_words_token_ids`, `cache_salt`, `logprob_token_ids` or `detokenize` in either block on the policy model: the frontend accepts them but drops them before NeMo RL's chat engine sees them.
+- `vllm_cfg.skip_tokenizer_init: true` with NeMo-Gym: the engine detokenizes the replies Gym's calls get through the frontend.
- `vllm_cfg.expert_parallel_size > tensor_parallel_size`: that runs data parallelism inside vLLM, which Dynamo does not serve yet. Above data-parallel rank 0, vLLM rejects each engine's `ipc://` KV-event endpoint.
- `vllm_cfg.http_server_api_key_required`: the frontend serves the same engines on 0.0.0.0 and checks no bearer token.
+- `vllm_cfg.http_api_server_count` above 1: NeMo-Gym's calls go through the Dynamo frontend, so the extra vLLM HTTP frontends would serve none of them. `frontend_args.render_workers` sets how many processes render chat calls behind Dynamo.
- `frontend_args.router_mode: direct`: it needs a worker ID on every request, and NeMo RL's rollouts name none.
- `dynamo: true` on a backend other than vLLM, and a `dynamo_cfg` block without `dynamo: true`.
@@ -86,7 +120,7 @@ bash docker/dynamo/install.sh
## Run
-Use any non-colocated SingleController vLLM recipe that does not use NeMo-Gym, and add the switch:
+Use any non-colocated SingleController vLLM recipe and add the switch:
```bash
uv run examples/run_grpo_single_controller.py \
diff --git a/examples/configs/grpo_math_1B_dynamo_single_controller.yaml b/examples/configs/grpo_math_1B_dynamo_single_controller.yaml
index a1d89235d1e..2c1ff204b1d 100644
--- a/examples/configs/grpo_math_1B_dynamo_single_controller.yaml
+++ b/examples/configs/grpo_math_1B_dynamo_single_controller.yaml
@@ -17,10 +17,16 @@ policy:
# Deadline for one rollout through the frontend. null waits as long as
# generation takes, as native generate_async does; on the SingleController,
# async_rl.rollout_failure.native.generation_timeout_s bounds both paths.
+ # NeMo-Gym's own calls keep Gym's timeouts.
request_timeout_s: null
frontend_args:
# kv: route each request to the engine holding most of its prefix.
router_mode: kv
+ # Keep a NeMo-Gym token-capture rollout on one engine while it idles up
+ # to this many seconds (null disables it).
+ session_affinity_ttl_secs: 300
+ # Processes that render NeMo-Gym's chat calls and build their replies.
+ render_workers: 8
extra_cli_args: []
logger:
diff --git a/examples/configs/recipes/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.yaml b/examples/configs/recipes/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.yaml
new file mode 100644
index 00000000000..56f5c8c74e6
--- /dev/null
+++ b/examples/configs/recipes/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.yaml
@@ -0,0 +1,180 @@
+defaults: ../../../nemo_gym/grpo_nanov3.yaml
+cluster:
+ num_nodes: 3
+ segment_size: 1
+checkpointing:
+ enabled: false
+ checkpoint_dir: results/nemotron-3-nano-swe-dynamo
+ save_period: 3
+ keep_top_k: 1
+ checkpoint_must_save_by: 00:03:30:00
+grpo:
+ num_prompts_per_step: 2
+ num_generations_per_prompt: 2
+ val_num_generations_per_prompt: 2
+ max_num_epochs: 4
+ max_num_steps: 4
+ advantage_clip_low: -100
+ advantage_clip_high: 100
+ val_period: 10000
+ overlong_filtering: false
+ invalid_tool_call_advantage: -5.0
+ malformed_thinking_advantage: -5.0
+ reward_shaping:
+ overlong_buffer_length: 128
+ overlong_buffer_penalty: 1
+ max_response_length: ${policy.max_total_sequence_length}
+ stop_properly_penalty_coef: null
+ reward_scaling:
+ source_min: 0.0
+ source_max: 1.0
+ target_min: 0.0
+ target_max: 1.0
+ async_grpo:
+ enabled: true
+ in_flight_weight_updates: false
+ recompute_kv_cache_after_weight_updates: true
+loss_fn:
+ truncated_importance_sampling_ratio: 5
+ truncated_importance_sampling_ratio_min: 0.2
+ truncated_importance_sampling_type: tis
+ force_on_policy_ratio: true
+ use_kl_in_reward: false
+policy:
+ model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
+ tokenizer:
+ name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
+ chat_template_kwargs:
+ enable_thinking: true
+ hf_config_overrides: {}
+ train_global_batch_size: 4
+ generation_batch_size: 2
+ max_total_sequence_length: 196608
+ logprob_chunk_size: 1024
+ megatron_cfg:
+ empty_unused_memory_level: 2
+ tensor_model_parallel_size: 4
+ pipeline_model_parallel_size: 1
+ context_parallel_size: 2
+ moe_flex_dispatcher_backend: hybridep
+ moe_hybridep_num_sms: 32
+ use_gloo_process_groups: false
+ use_fused_weighted_squared_relu: true
+ do_not_average_loss: true
+ cp_normalize: true
+ calculate_per_token_loss: true
+ scale_loss_by_dp_cp_size: false
+ mtp_loss_scaling_factor: 0.0
+ mtp_use_repeated_layer: true
+ mtp_num_layers: 0
+ mtp_detach_heads: true
+ scheduler:
+ override_opt_param_scheduler: true
+ distributed_data_parallel_config:
+ overlap_grad_reduce: false
+ fp8_cfg:
+ enabled: false
+ fp8: e4m3
+ fp8_recipe: mxfp8
+ fp8_param: false
+ checkpoint:
+ ckpt_assume_constant_structure: true
+ sequence_packing:
+ fuse_loss: true
+ generation:
+ dynamo: true
+ dynamo_cfg:
+ startup_timeout_s: 1800
+ frontend_args:
+ router_mode: kv
+ extra_cli_args: []
+ vllm_cfg:
+ expert_parallel_size: 4
+ precision: bfloat16
+ gpu_memory_utilization: 0.85
+ enable_vllm_metrics_logger: true
+ vllm_metrics_logger_interval: 0.5
+ env_vars: null
+ vllm_kwargs:
+ attention_backend: FLASH_ATTN
+ moe_backend: triton
+ compilation_config:
+ cudagraph_capture_sizes:
+ - 1
+ - 2
+ - 4
+ - 8
+ - 16
+ - 32
+ - 64
+ pass_config:
+ fuse_allreduce_rms: false
+ colocated:
+ enabled: false
+ resources:
+ gpus_per_node: 8
+ num_nodes: 1
+data:
+ train:
+ data_path: ${oc.env:HF_HOME}/superv3_data/swe1/train-split.jsonl
+ validation:
+ data_path: ${oc.env:HF_HOME}/superv3_data/swe1/val-split.jsonl
+env:
+ nemo_gym:
+ skip_venv_if_present: true
+ num_gpu_nodes: 0
+ invalid_tool_call_patterns:
+ -
+ -
+ -
+ -
+ thinking_tags:
+ -
+ -
+ config_paths:
+ - responses_api_models/vllm_model/configs/vllm_model_for_training.yaml
+ - resources_servers/single_step_tool_use_with_argument_comparison/configs/swe_pivot_single_step_tool_use_with_argument_comparison.yaml
+ # Drop inherited wrappers for environments omitted from this recipe.
+ math_with_judge_environment_server: null
+ code_gen_environment_server: null
+ workplace_assistant_environment_server: null
+ mcqa_environment_server: null
+ instruction_following_environment_server: null
+ structured_outputs_environment_server: null
+ swe_pivot_single_step_tool_use_with_argument_comparison_environment_server:
+ environment_servers:
+ legacy_agent:
+ entrypoint: app.py
+ agent_server:
+ type: responses_api_agents
+ name: swe_pivot_single_step_tool_use_with_argument_comparison_agent
+ policy_model:
+ responses_api_models:
+ vllm_model:
+ chat_template_kwargs:
+ force_nonempty_content: true
+ single_step_tool_use_with_argument_comparison_swe:
+ responses_api_agents:
+ tool_simulation_agent:
+ entrypoint: app.py
+ resources_server:
+ type: resources_servers
+ name: swe_pivot_single_step_tool_use_with_argument_comparison_resources_server
+ model_server:
+ type: responses_api_models
+ name: policy_model
+ single_step_tool_use_with_argument_comparison_swe_environment_server:
+ environment_servers:
+ legacy_agent:
+ entrypoint: app.py
+ agent_server:
+ type: responses_api_agents
+ name: single_step_tool_use_with_argument_comparison_swe
+ use_absolute_ip: true
+logger:
+ log_dir: results/nemotron-3-nano-swe-dynamo
+ wandb_enabled: true
+ tensorboard_enabled: true
+ wandb:
+ project: nemo-rl
+ name: grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1
diff --git a/examples/modelopt/qa_distillation_nano3_megatron.yaml b/examples/modelopt/qa_distillation_nano3_megatron.yaml
index bfe75b2844d..a8cfbb6b9de 100644
--- a/examples/modelopt/qa_distillation_nano3_megatron.yaml
+++ b/examples/modelopt/qa_distillation_nano3_megatron.yaml
@@ -158,7 +158,7 @@ env:
nemo_gym:
config_paths:
- responses_api_models/vllm_model/configs/vllm_model_for_training.yaml
- - responses_api_agents/verifiers_agent/configs/acereason-math.yaml
+ - responses_api_agents/verifiers_agent/configs/verifiers_agent.yaml
# Use legacy agent orchestration.
verifiers_agent_environment_server:
diff --git a/examples/nemo_gym/run_distillation_nemo_gym.py b/examples/nemo_gym/run_distillation_nemo_gym.py
index 1520c0da4a0..03592b83461 100644
--- a/examples/nemo_gym/run_distillation_nemo_gym.py
+++ b/examples/nemo_gym/run_distillation_nemo_gym.py
@@ -109,7 +109,7 @@ def main() -> None:
# We assert here since this is right after the final config has been materialized.
assert should_use_nemo_gym(config)
- # NeMo-Gym environment needs to get dp_openai_server_base_urls from
+ # NeMo-Gym environment needs to get client_base_urls from
# student_generation, so we don't setup env here.
print("\n▶ Setting up data...")
train_dataset, val_dataset = setup_response_data(
diff --git a/examples/nemo_gym/run_grpo_nemo_gym.py b/examples/nemo_gym/run_grpo_nemo_gym.py
index 6ac863f2ab2..b1504fefb04 100644
--- a/examples/nemo_gym/run_grpo_nemo_gym.py
+++ b/examples/nemo_gym/run_grpo_nemo_gym.py
@@ -209,7 +209,7 @@ def main() -> None:
config.grpo.val_period > 0 or config.grpo.val_at_start or config.grpo.val_at_end
)
- # NeMo-Gym environment needs to get dp_openai_server_base_urls from policy_generation, so we don't setup env here.
+ # NeMo-Gym environment needs to get client_base_urls from policy_generation, so we don't setup env here.
with rl_init_timer.time("data"):
print("\n▶ Setting up data...")
data_tokenizer = processor if processor is not None else tokenizer
diff --git a/nemo_rl/algorithms/distillation.py b/nemo_rl/algorithms/distillation.py
index 065352cbbc7..7c936ed28a8 100644
--- a/nemo_rl/algorithms/distillation.py
+++ b/nemo_rl/algorithms/distillation.py
@@ -562,7 +562,7 @@ def init_vllm_deferred():
def init_nemo_gym():
shard_set = build_nemo_gym_actors(
env_configs,
- base_urls=cast(list[str], deferred_vllm.dp_openai_server_base_urls),
+ base_urls=cast(list[str], deferred_vllm.client_base_urls),
model_name=generation_config["model_name"],
tokenizer=tokenizer,
# Distillation does not configure vLLM for router replay.
diff --git a/nemo_rl/algorithms/grpo.py b/nemo_rl/algorithms/grpo.py
index e41b5766605..c71aaad0150 100644
--- a/nemo_rl/algorithms/grpo.py
+++ b/nemo_rl/algorithms/grpo.py
@@ -1829,8 +1829,8 @@ def init_megatron_generation_task(policy_future):
)
vllm_reserve_time = time.perf_counter() - vllm_reserve_t0
print(
- f" ✓ Reserved {len(deferred_vllm.dp_openai_server_base_urls)} vLLM server URLs: "
- f"{deferred_vllm.dp_openai_server_base_urls}",
+ f" ✓ Reserved {len(deferred_vllm.client_base_urls)} vLLM server URLs: "
+ f"{deferred_vllm.client_base_urls}",
flush=True,
)
@@ -1844,7 +1844,7 @@ def init_vllm_deferred():
def init_nemo_gym():
"""Spin up NeMo Gym servers with the pre-assigned vLLM URLs."""
return _spinup_nemo_gym(
- deferred_vllm.dp_openai_server_base_urls,
+ deferred_vllm.client_base_urls,
generation_config["model_name"],
policy_api_key=deferred_vllm.http_server_api_key,
)
diff --git a/nemo_rl/algorithms/single_controller_utils/config.py b/nemo_rl/algorithms/single_controller_utils/config.py
index 8819c6b9284..47776d23f2c 100644
--- a/nemo_rl/algorithms/single_controller_utils/config.py
+++ b/nemo_rl/algorithms/single_controller_utils/config.py
@@ -726,9 +726,12 @@ class TokenCaptureConfig(BaseModel, extra="allow"):
# derived at setup
# under the run's log dir.
capture_dir: Optional[str] = None
- # Generation backend hosting token capture. This is derived from
- # policy.generation.backend during setup; users should not set it separately.
- generation_backend: Optional[Literal["vllm", "megatron"]] = None
+ # How NeMo-Gym reaches the capture host; derived during setup, users should not
+ # set it. "dynamo" = vLLM served through the Dynamo frontend
+ # (policy.generation.dynamo): same workers and capture code, but Gym's
+ # dynamo_worker handler sends capture fields in nvext, because the frontend
+ # carries only OpenAI fields plus nvext.
+ generation_backend: Optional[Literal["vllm", "megatron", "dynamo"]] = None
# Keep routed_experts out of canonical rows and assemble them on policy
# workers from strict staged-fragment plans.
defer_routed_experts_to_policy: bool = False
diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py
index 4549ac57ad6..83aa0620dce 100644
--- a/nemo_rl/algorithms/single_controller_utils/setup.py
+++ b/nemo_rl/algorithms/single_controller_utils/setup.py
@@ -133,6 +133,7 @@
should_mask_flagged_samples,
)
from nemo_rl.models.generation import resolve_generation_class
+from nemo_rl.models.generation.dynamo.config import dynamo_enabled
from nemo_rl.models.generation.fleet_health import (
FleetHealthPolicy,
GenerationFleetHealth,
@@ -801,6 +802,7 @@ def _spinup_gym(
base_urls: list[str],
tokenizer: PreTrainedTokenizerBase,
policy_api_key: Optional[str],
+ generation_cut_control_urls: Optional[list[str]],
) -> tuple[NemoGymShardSet, float]:
"""Spin up the NeMo-Gym shard set against the reserved vLLM URLs.
@@ -811,6 +813,9 @@ def _spinup_gym(
call. See NemoGym.set_tokenizer.
policy_api_key: The bearer token the vLLM server requires on its OpenAI
routes (VllmGeneration.http_server_api_key), or None when it checks none.
+ generation_cut_control_urls: Where Gym sends checkpoint generation cuts,
+ one per base URL (VllmGeneration.generation_cut_control_urls), or
+ None when each base URL takes its own.
Returns:
A tuple of (NeMo-Gym shard set, wall time spent in this call).
@@ -833,6 +838,7 @@ def _spinup_gym(
generation_prefix_cuts_enabled=(
master_config.rollout_recovery.generation_prefix_cuts_enabled
),
+ generation_cut_control_urls=generation_cut_control_urls,
# Capture transport config includes the generation request's candidate count.
token_capture=(
{
@@ -1455,7 +1461,11 @@ def setup_single_controller(
# Fill the derived ledger-hosting fields (see TokenCaptureConfig): a
# per-run control-plane bearer token, the process-shared capture
# directory used by every Gym worker, and the capture-host backend.
- token_capture_cfg.generation_backend = generation_config["backend"]
+ token_capture_cfg.generation_backend = (
+ "dynamo"
+ if dynamo_enabled(generation_config)
+ else generation_config["backend"]
+ )
if token_capture_cfg.control_auth_token is None:
# Deferred import: only needed on the capture path.
import secrets
@@ -1962,7 +1972,7 @@ def _build_trainer_then_megatron_generation() -> tuple[
defer_model_load=True,
)
defer_generation_model_load = True
- gym_base_urls = generation.dp_openai_server_base_urls
+ gym_base_urls = generation.client_base_urls
# Before the Gym task is built, so Gym can be handed the router's single URL.
# These two statements are the only failable ones related to the port holder's creation.
try:
@@ -1989,6 +1999,11 @@ def _build_trainer_then_megatron_generation() -> tuple[
policy_api_key=(
generation.http_server_api_key if generation is not None else None
),
+ generation_cut_control_urls=(
+ generation.generation_cut_control_urls
+ if generation is not None
+ else None
+ ),
)
if megatron_backend and colocated:
diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py
index e26a5cb46bb..def7823cab0 100644
--- a/nemo_rl/environments/nemo_gym.py
+++ b/nemo_rl/environments/nemo_gym.py
@@ -126,7 +126,9 @@
NEMO_GYM_GROUP_ID_KEY,
NEMO_GYM_ROLLOUT_INDEX_KEY,
)
-from nemo_rl.models.generation.dynamo.config import dynamo_enabled
+from nemo_rl.models.generation.dynamo.config import (
+ dynamo_enabled,
+)
from nemo_rl.models.generation.interfaces import (
resolve_routed_experts_dtype_name_for_model,
should_use_async_rollouts,
@@ -485,12 +487,6 @@ def should_use_nemo_gym(master_config: NemoGymCompatibleConfig) -> bool:
return False
generation_config = master_config.policy["generation"]
- if dynamo_enabled(generation_config):
- raise ValueError(
- "policy.generation.dynamo=true does not serve NeMo-Gym rollouts yet. "
- "Set policy.generation.dynamo=false to run this NeMo-Gym config on "
- "NeMo RL's vLLM servers."
- )
assert should_use_async_rollouts(generation_config), (
"❌ Error: In order to use NeMo-Gym, you must use a generation "
"backend with `async_engine: true`!"
@@ -804,9 +800,11 @@ def _external_staging_backend(token_capture: Dict[str, Any]) -> str:
return "vllm_worker"
if generation_backend == "megatron":
return "megatron_worker"
+ if generation_backend == "dynamo":
+ return "dynamo_worker"
raise ValueError(
"token_capture.enabled requires setup-derived generation_backend to be "
- f"'vllm' or 'megatron'; got {generation_backend!r}"
+ f"'vllm', 'megatron' or 'dynamo'; got {generation_backend!r}"
)
@@ -1354,7 +1352,8 @@ def _spinup(self) -> None:
)
policy_overrides["return_token_id_information"] = False
if (
- _external_staging_backend(token_capture) == "vllm_worker"
+ _external_staging_backend(token_capture)
+ in ("vllm_worker", "dynamo_worker")
and token_capture.get("top_logprobs", 0) > 0
):
policy_overrides.setdefault("sampling_overrides", {}).update(
@@ -2942,6 +2941,12 @@ def setup_nemo_gym_config(config, tokenizer) -> None:
generation_config["stop_strings"] = None
generation_config["stop_token_ids"] = None
+ if dynamo_enabled(generation_config):
+ # Gym's policy model calls the Dynamo frontend in Gym's Dynamo dialect.
+ # Each actor applies it to its own composed config (_apply_dynamo_dialect),
+ # so it never becomes a policy_model overlay that a shard must claim.
+ config.env.setdefault("nemo_gym", {})["dynamo_dialect"] = True
+
# For VLM runs, plumb the tokenizer config into the gym env config so the
# NemoGym actor can reconstruct the processor inside itself (needed for
# multi-turn multimodal postprocessing).
@@ -2959,6 +2964,7 @@ def build_nemo_gym_config(
use_fastokens: bool,
turn_checkpointing_enabled: bool,
generation_prefix_cuts_enabled: bool = False,
+ generation_cut_control_urls: Optional[list[str]] = None,
token_capture: Optional[dict[str, Any]] = None,
policy_api_key: Optional[str] = None,
) -> NemoGymConfig:
@@ -2983,6 +2989,10 @@ def build_nemo_gym_config(
generation_prefix_cuts_enabled: Let Gym cut active model calls at a
checkpoint so a restore resumes them from their decoded prefix.
Requires ``turn_checkpointing_enabled``.
+ generation_cut_control_urls: Where Gym sends those cuts, one per entry
+ of ``base_urls``; None when each base URL takes its own, as a
+ vLLM server does. A Dynamo frontend takes them on another port
+ (VllmGeneration.generation_cut_control_urls).
policy_api_key: The bearer token the generation server requires on its
OpenAI routes (VllmGeneration.http_server_api_key), or None when it
checks none.
@@ -3018,11 +3028,59 @@ def build_nemo_gym_config(
),
turn_checkpointing_enabled=turn_checkpointing_enabled,
generation_prefix_cuts_enabled=generation_prefix_cuts_enabled,
+ generation_cut_control_urls=generation_cut_control_urls,
token_capture=token_capture,
policy_api_key=policy_api_key,
)
+def _apply_dynamo_dialect(nemo_gym_dict: dict[str, Any]) -> None:
+ """Point Gym's `policy_model` server in one NemoGym actor's config at the Dynamo frontend.
+
+ The frontend takes token data in nvext instead of vLLM's request extensions.
+ Applied to the actor's composed config (a shard's overrides included), so
+ settings the user wrote there win. Sets ``policy_model`` on ``nemo_gym_dict``
+ in place, so pass a per-actor copy; the nested policy_model levels are
+ copied, so dicts shared with the user's config are not mutated.
+ """
+ policy_model = dict(nemo_gym_dict.get("policy_model") or {})
+ models = dict(policy_model.get("responses_api_models") or {})
+ vllm_model = dict(models.get("vllm_model") or {})
+ # Dynamo accepts these vLLM fields (its PASSTHROUGH_EXTRA_FIELDS) but never
+ # serializes them to a Python chat engine, so ours would never see them.
+ dropped_by_frontend = {
+ "allowed_token_ids",
+ "bad_words_token_ids",
+ "cache_salt",
+ "detokenize",
+ "logprob_token_ids",
+ "stop_token_ids",
+ }
+ # Gym puts the keys of both blocks on every request as top-level fields.
+ for block in ("extra_body", "sampling_overrides"):
+ fields = vllm_model.get(block) or {}
+ if "bad_words" in fields:
+ raise ValueError(
+ f"vllm_model.{block}.bad_words is not supported with "
+ "policy.generation.dynamo=true: the Dynamo frontend answers every "
+ "request carrying it with HTTP 400."
+ )
+ dropped = sorted(dropped_by_frontend & fields.keys())
+ if dropped:
+ raise ValueError(
+ f"vllm_model.{block} {dropped} is not supported with "
+ "policy.generation.dynamo=true: the Dynamo frontend accepts these "
+ "fields but drops them before NeMo RL's chat engine sees them."
+ )
+ vllm_model["dynamo_frontend"] = True
+ # The frontend reads `reasoning` as another name for `reasoning_content` and
+ # refuses an echoed assistant turn that carries both, Gym's default.
+ vllm_model.setdefault("reasoning_field", "reasoning_content")
+ models["vllm_model"] = vllm_model
+ policy_model["responses_api_models"] = models
+ nemo_gym_dict["policy_model"] = policy_model
+
+
def _build_gym_actor_config(
nemo_gym_dict: dict[str, Any],
*,
@@ -3033,6 +3091,7 @@ def _build_gym_actor_config(
checkpoint_instance: GymCheckpointInstance,
turn_checkpointing_enabled: bool,
generation_prefix_cuts_enabled: bool,
+ generation_cut_control_urls: Optional[list[str]],
token_capture: Optional[dict[str, Any]] = None,
policy_api_key: Optional[str] = None,
) -> NemoGymConfig:
@@ -3054,6 +3113,8 @@ def _build_gym_actor_config(
invalid_tool_call_patterns = nemo_gym_dict.pop("invalid_tool_call_patterns", None)
thinking_tags = nemo_gym_dict.pop("thinking_tags", None)
tokenizer_config = nemo_gym_dict.pop("tokenizer_config", None)
+ if nemo_gym_dict.pop("dynamo_dialect", False):
+ _apply_dynamo_dialect(nemo_gym_dict)
external_service_readiness = extract_external_service_readiness(nemo_gym_dict)
port_range = {
key: value
@@ -3126,6 +3187,11 @@ def _build_gym_actor_config(
policy_overrides["checkpoint"] = {"policy": True}
if generation_prefix_cuts_enabled:
policy_overrides["checkpoint"]["generation_cuts"] = True
+ if generation_cut_control_urls is not None:
+ # The base URLs do not take cuts themselves (a Dynamo frontend).
+ policy_overrides["generation_cut_control_url"] = list(
+ generation_cut_control_urls
+ )
return NemoGymConfig(
model_name=model_name,
@@ -3430,6 +3496,7 @@ def build_nemo_gym_actors(
use_fastokens: bool,
turn_checkpointing_enabled: bool,
generation_prefix_cuts_enabled: bool = False,
+ generation_cut_control_urls: Optional[list[str]] = None,
token_capture: Optional[dict[str, Any]] = None,
policy_api_key: Optional[str] = None,
pg_ready_timeout: float = DEFAULT_SHARD_PG_READY_TIMEOUT_SECONDS,
@@ -3451,6 +3518,10 @@ def build_nemo_gym_actors(
generation_prefix_cuts_enabled: Let Gym cut active model calls at a
checkpoint so a restore resumes them from their decoded prefix.
Requires ``turn_checkpointing_enabled``.
+ generation_cut_control_urls: Where Gym sends those cuts, one per entry
+ of ``base_urls``; None when each base URL takes its own, as a
+ vLLM server does. A Dynamo frontend takes them on another port
+ (VllmGeneration.generation_cut_control_urls).
Returns:
A :class:`NemoGymShardSet` whose actors are all running and validated.
@@ -3473,6 +3544,7 @@ def build_nemo_gym_actors(
use_fastokens=use_fastokens,
turn_checkpointing_enabled=turn_checkpointing_enabled,
generation_prefix_cuts_enabled=generation_prefix_cuts_enabled,
+ generation_cut_control_urls=generation_cut_control_urls,
token_capture=token_capture,
policy_api_key=policy_api_key,
)
@@ -3487,6 +3559,7 @@ def build_nemo_gym_actors(
use_fastokens=use_fastokens,
turn_checkpointing_enabled=turn_checkpointing_enabled,
generation_prefix_cuts_enabled=generation_prefix_cuts_enabled,
+ generation_cut_control_urls=generation_cut_control_urls,
token_capture=token_capture,
policy_api_key=policy_api_key,
pg_ready_timeout=pg_ready_timeout,
@@ -3504,6 +3577,7 @@ def _build_single_gym_actor(
use_fastokens: bool,
turn_checkpointing_enabled: bool,
generation_prefix_cuts_enabled: bool,
+ generation_cut_control_urls: Optional[list[str]],
token_capture: Optional[dict[str, Any]],
policy_api_key: Optional[str] = None,
) -> NemoGymShardSet:
@@ -3520,6 +3594,7 @@ def _build_single_gym_actor(
),
turn_checkpointing_enabled=turn_checkpointing_enabled,
generation_prefix_cuts_enabled=generation_prefix_cuts_enabled,
+ generation_cut_control_urls=generation_cut_control_urls,
token_capture=token_capture,
policy_api_key=policy_api_key,
)
@@ -3570,6 +3645,7 @@ def _build_sharded_gym_actors(
use_fastokens: bool,
turn_checkpointing_enabled: bool,
generation_prefix_cuts_enabled: bool,
+ generation_cut_control_urls: Optional[list[str]],
token_capture: Optional[dict[str, Any]],
policy_api_key: Optional[str] = None,
pg_ready_timeout: float,
@@ -3660,6 +3736,7 @@ def _build_sharded_gym_actors(
),
turn_checkpointing_enabled=turn_checkpointing_enabled,
generation_prefix_cuts_enabled=generation_prefix_cuts_enabled,
+ generation_cut_control_urls=generation_cut_control_urls,
token_capture=token_capture,
policy_api_key=policy_api_key,
)
diff --git a/nemo_rl/environments/nemo_gym_shards.py b/nemo_rl/environments/nemo_gym_shards.py
index 8a0a3eccaef..84771acef99 100644
--- a/nemo_rl/environments/nemo_gym_shards.py
+++ b/nemo_rl/environments/nemo_gym_shards.py
@@ -38,7 +38,11 @@
# Dict-shaped NeMo RL integration settings that are not Gym server entries.
# The actor factory or rollout code consumes these separately.
NEMO_RL_DICT_CONFIG_KEYS = frozenset(
- {"effort_levels", "external_service_readiness", "tokenizer_config"}
+ {
+ "effort_levels",
+ "external_service_readiness",
+ "tokenizer_config",
+ }
)
# Gym settings that contain mappings but are not server-instance overlays.
diff --git a/nemo_rl/models/generation/__init__.py b/nemo_rl/models/generation/__init__.py
index b5fb07139f2..0d682cae973 100644
--- a/nemo_rl/models/generation/__init__.py
+++ b/nemo_rl/models/generation/__init__.py
@@ -223,6 +223,14 @@ def configure_generation_config(
"frontend serves the same engines on 0.0.0.0 and checks no "
"bearer token."
)
+ if config["vllm_cfg"].get("http_api_server_count", 1) > 1:
+ raise ValueError(
+ "policy.generation.vllm_cfg.http_api_server_count > 1 is not "
+ "supported with policy.generation.dynamo=true: NeMo-Gym's calls "
+ "go through the Dynamo frontend, so the extra vLLM HTTP "
+ "frontends would serve none of them. Size "
+ "dynamo_cfg.frontend_args.render_workers instead."
+ )
colocated = config.get("colocated")
if colocated is not None and colocated["enabled"]:
raise ValueError(
diff --git a/nemo_rl/models/generation/capture_context.py b/nemo_rl/models/generation/capture_context.py
index 8a9673737a9..c252e9c7ec1 100644
--- a/nemo_rl/models/generation/capture_context.py
+++ b/nemo_rl/models/generation/capture_context.py
@@ -94,3 +94,34 @@ def decide_capture_input(
generation_cut=admission.generation_cut,
)
return CaptureInputDecision(serving, serving, True)
+
+
+def resolve_capture_input(
+ admission: "CaptureAdmission",
+ *,
+ messages: list[dict[str, Any]],
+ conversation: list[dict[str, Any]],
+ renderer_is_hf: bool,
+) -> CaptureInputDecision:
+ """:func:`decide_capture_input` with the rendering fact the serving host proves.
+
+ Shared by every host of the Gym chat path, which calls it after rendering
+ the conversation. HF/string parsing maps null and empty tool content to
+ identical template inputs; structured content and other renderers cannot
+ authorize that equivalence.
+
+ Args:
+ admission: The call's capture admission.
+ messages: The request's messages, before rendering.
+ conversation: The rendered conversation the template saw.
+ renderer_is_hf: The host renders with vLLM's ``HfRenderer``.
+ """
+ return decide_capture_input(
+ admission,
+ messages=messages,
+ allow_empty_tool_content=(
+ renderer_is_hf
+ and bool(conversation)
+ and all(isinstance(item.get("content"), str) for item in conversation)
+ ),
+ )
diff --git a/nemo_rl/models/generation/dynamo/config.py b/nemo_rl/models/generation/dynamo/config.py
index 824280c3edf..f8cc16dfed5 100644
--- a/nemo_rl/models/generation/dynamo/config.py
+++ b/nemo_rl/models/generation/dynamo/config.py
@@ -15,13 +15,14 @@
"""Configuration for serving NeMo RL's vLLM workers behind Dynamo.
``policy.generation.dynamo: true`` on any vLLM config is the whole switch;
-``policy.generation.dynamo_cfg`` only tunes the frontend and registration, and
-every field below carries its default.
+``policy.generation.dynamo_cfg`` only tunes the frontend, and every field below
+carries its default. NeMo-Gym's chat calls are rendered and parsed with the
+native server's own settings (``vllm_cfg.http_server_serving_chat_kwargs``).
"""
-from typing import Any, Literal, Mapping
+from typing import Annotated, Any, Literal, Mapping
-from pydantic import BaseModel, Field, PositiveFloat
+from pydantic import BaseModel, Field, PositiveFloat, PositiveInt, model_validator
class DynamoFrontendArgs(BaseModel, extra="forbid"):
@@ -38,17 +39,41 @@ class DynamoFrontendArgs(BaseModel, extra="forbid"):
"least-loaded",
"device-aware-weighted",
] = "kv"
+ # Idle seconds a NeMo-Gym token-capture session stays on one engine, as native
+ # pins each Gym session to one server; None disables it. Needs Dynamo 1.7
+ # (ai-dynamo/dynamo#15333): 1.5.1 drops a chat call's session id. Dynamo keeps
+ # at most 65,536 sessions (HTTP 529 past that).
+ session_affinity_ttl_secs: Annotated[int, Field(gt=0, le=31_536_000)] | None = 300
+ # Processes that render NeMo-Gym's chat calls and build their replies (the
+ # checkpoint's template, tokenizer, HF processor and vLLM's parsers); each
+ # uses one core. See docs/guides/dynamo-generation.md for sizing.
+ render_workers: PositiveInt = 8
extra_cli_args: list[str] = Field(default_factory=list)
+ @model_validator(mode="after")
+ def _chat_is_nemo_rls(self) -> "DynamoFrontendArgs":
+ # NeMo-Gym's chat calls are rendered and parsed by NeMo RL's chat
+ # engine, never by Dynamo's own renderer or chat processor.
+ for flag in ("--dyn-chat-processor", "--custom-jinja-template"):
+ if any(arg.split("=", 1)[0] == flag for arg in self.extra_cli_args):
+ raise ValueError(
+ f"dynamo_cfg.frontend_args.extra_cli_args may not set {flag}: "
+ "NeMo RL's chat engine renders and parses chat calls with "
+ "vllm_cfg.http_server_serving_chat_kwargs, as the native server "
+ "does."
+ )
+ return self
+
class DynamoCfg(BaseModel, extra="forbid"):
- """``policy.generation.dynamo_cfg``: the frontend and engine registration."""
+ """``policy.generation.dynamo_cfg``: the managed frontend."""
# Bounds etcd/NATS/frontend start-up and every engine's registration.
startup_timeout_s: PositiveFloat = 600.0
# Deadline for one NeMo RL rollout through the frontend. None waits as long
# as generation takes, as native generate_async does; on the SingleController,
# async_rl.rollout_failure.native.generation_timeout_s bounds both paths.
+ # NeMo-Gym's own calls keep Gym's timeouts.
request_timeout_s: PositiveFloat | None = None
frontend_args: DynamoFrontendArgs = Field(default_factory=DynamoFrontendArgs)
diff --git a/nemo_rl/models/generation/dynamo/engine_adapter.py b/nemo_rl/models/generation/dynamo/engine_adapter.py
index 40ae920be17..517f6a0d371 100644
--- a/nemo_rl/models/generation/dynamo/engine_adapter.py
+++ b/nemo_rl/models/generation/dynamo/engine_adapter.py
@@ -18,8 +18,9 @@
request kinds served, how errors reach the driver, and the endpoint's
registration. A backend implements four methods over its worker's engine
(``serve_rollout``, ``start``, ``abort`` and ``kv_event_sources``);
-``vllm_adapter.DynamoVllmAdapter`` is the vLLM one. The engine only ever receives
-token IDs, on ``/v1/completions``. This module imports no generation backend.
+``vllm.adapter.DynamoVllmAdapter`` is the vLLM one. The engine only ever receives
+token IDs: NeMo RL's rollouts on ``/v1/completions``, and NeMo-Gym's chat calls
+once NeMo RL's chat engine in the frontend has rendered them. This module imports no generation backend.
"""
from __future__ import annotations
@@ -32,7 +33,7 @@
from dataclasses import dataclass
from typing import Any, Optional
-from dynamo._core import Context
+from dynamo._core import Context, fetch_model
from dynamo.common.backend import (
EngineConfig,
GenerateChunk,
@@ -114,7 +115,7 @@ async def generate(
break
else:
raise InvalidArgument(
- "NeMo RL's engines serve only NeMo RL rollouts through Dynamo "
+ "NeMo RL's engines serve only NeMo RL's own requests through Dynamo "
f"(requests annotated {', '.join(map(repr, handlers))})."
)
try:
@@ -180,6 +181,17 @@ async def cleanup(self) -> None:
"""Nothing to release: the worker owns its engine and shuts it down."""
+async def weight_free_model_path(model: str) -> str:
+ """A local directory with ``model``'s config and tokenizer files.
+
+ ``model`` itself when it is a local path; for a Hub ID, Dynamo's snapshot
+ of it without the weights.
+ """
+ if os.path.exists(model):
+ return model
+ return await fetch_model(model, ignore_weights=True)
+
+
def start_dynamo_endpoint(
adapter: DynamoEngineAdapter,
*,
@@ -191,7 +203,9 @@ def start_dynamo_endpoint(
"""Register ``adapter``'s engine with Dynamo and serve it on the running loop.
``model_path`` is a local directory with the model's config and tokenizer
- files; the engine is served under ``served_model_name``.
+ files; the engine is served under ``served_model_name``. Chat calls are
+ rendered by NeMo RL's chat engine in the frontend, not Dynamo's renderer,
+ so no template or parser is registered.
The Rust Worker serves until the process gets SIGTERM. It installs a chained
handler, so Ray's actor handler still runs and the actor exits as usual;
@@ -207,8 +221,8 @@ def start_dynamo_endpoint(
endpoint=DYNAMO_ENDPOINT,
model_name=model_path,
served_model_name=served_model_name,
- # NeMo RL's rollouts go to /v1/completions only.
- endpoint_types="completions",
+ # NeMo RL's rollouts go to /v1/completions; NeMo-Gym calls chat.
+ endpoint_types="chat,completions",
discovery_backend=engine_env["DYN_DISCOVERY_BACKEND"],
request_plane=engine_env["DYN_REQUEST_PLANE"],
event_plane=engine_env["DYN_EVENT_PLANE"],
diff --git a/nemo_rl/models/generation/dynamo/managed_runtime.py b/nemo_rl/models/generation/dynamo/managed_runtime.py
index be438e817b0..9c6f7d95ac7 100644
--- a/nemo_rl/models/generation/dynamo/managed_runtime.py
+++ b/nemo_rl/models/generation/dynamo/managed_runtime.py
@@ -22,6 +22,7 @@
import json
import logging
import os
+import pickle
import re
import shutil
import signal
@@ -31,7 +32,7 @@
import time
import urllib.error
import urllib.request
-from typing import Callable
+from typing import Any, Callable
from nemo_rl.distributed.virtual_cluster import (
DEFAULT_DYNAMO_CONTROL_PORT_RANGE_HIGH,
@@ -70,6 +71,11 @@ def frontend_argv(
"--router-mode",
dynamo_cfg.frontend_args.router_mode,
]
+ if dynamo_cfg.frontend_args.session_affinity_ttl_secs is not None:
+ argv += [
+ "--router-session-affinity-ttl-secs",
+ str(dynamo_cfg.frontend_args.session_affinity_ttl_secs),
+ ]
extra = dynamo_cfg.frontend_args.extra_cli_args
for token in extra:
flag = token.split("=", 1)[0].replace("_", "-")
@@ -113,11 +119,49 @@ def _managed_namespace() -> str:
return re.sub(r"[^a-zA-Z0-9_-]+", "-", raw).strip("-_").lower()
+# The frontend process's environment for NeMo RL's chat engine
+# (dynamo/vllm/chat_engine.py): its render workers, where setup_token_capture
+# writes the capture config, the Ray cluster its TQ reads go through, and the
+# port it takes NeMo-Gym's generation cuts on (dynamo/vllm/generation_cut.py).
+RENDER_WORKERS_ENV = "NEMO_RL_DYNAMO_RENDER_WORKERS"
+# Where the driver writes the chat settings an engine sends before it loads.
+# Set only for a run that serves chat: the frontend then starts its render
+# workers at launch.
+CHAT_SETTINGS_FILE_ENV = "NEMO_RL_DYNAMO_CHAT_SETTINGS_FILE"
+CAPTURE_FILE_ENV = "NEMO_RL_DYNAMO_CAPTURE_FILE"
+RAY_ADDRESS_ENV = "NEMO_RL_DYNAMO_RAY_ADDRESS"
+GENERATION_CUT_PORT_ENV = "NEMO_RL_DYNAMO_GENERATION_CUT_PORT"
+
+
+def read_chat_capture() -> dict[str, Any] | None:
+ """What ``publish_chat_capture`` wrote for this frontend; None before it has."""
+ path = os.environ[CAPTURE_FILE_ENV]
+ if not os.path.exists(path):
+ return None
+ with open(path, "rb") as capture_file:
+ return pickle.load(capture_file)
+
+
+def read_chat_settings() -> dict[str, Any] | None:
+ """What ``publish_chat_settings`` wrote for this frontend; None before it has."""
+ path = os.environ[CHAT_SETTINGS_FILE_ENV]
+ if not os.path.exists(path):
+ return None
+ with open(path) as settings_file:
+ return json.load(settings_file)
+
+
class ManagedDynamoRuntime:
"""Own etcd, NATS and the frontend that NeMo RL's vLLM engines register with."""
def __init__(
- self, *, dynamo_cfg: DynamoCfg, model_name: str, frontend_python: str
+ self,
+ *,
+ dynamo_cfg: DynamoCfg,
+ model_name: str,
+ frontend_python: str,
+ ray_address: str,
+ serves_chat: bool,
) -> None:
"""Describe the services; nothing starts until :meth:`start`.
@@ -126,16 +170,25 @@ def __init__(
model_name: The served model; the frontend is ready once it lists it.
frontend_python: An interpreter whose environment has ``ai-dynamo``:
the vLLM workers' own.
+ ray_address: The Ray cluster's GCS address; the frontend's chat
+ engine reads token-capture prefixes from TQ through it.
+ serves_chat: Whether the engines expose chat
+ (``vllm_cfg.expose_http_server``, which NeMo-Gym requires). The
+ frontend then starts its render workers at launch.
"""
self._dynamo_cfg = dynamo_cfg
self._model_name = model_name
self._frontend_python = frontend_python
+ self._ray_address = ray_address
+ self.serves_chat = serves_chat
+ self._chat_dir: str | None = None
self._namespace = _managed_namespace()
self._host = ""
self._etcd_port = 0
self._etcd_peer_port = 0
self._nats_port = 0
self._frontend_port = 0
+ self._generation_cut_port = 0
self._manager_env: dict[str, str] = {}
self._started = False
self._etcd_process: subprocess.Popen | None = None
@@ -155,13 +208,21 @@ def frontend_url(self) -> str:
host = f"[{self._host}]" if ":" in self._host else self._host
return f"http://{host}:{self._frontend_port}/v1"
+ @property
+ def generation_cut_control_url(self) -> str:
+ """Where the frontend process takes NeMo-Gym's checkpoint generation cuts."""
+ if not self._started:
+ raise RuntimeError("Managed Dynamo runtime has not been started")
+ host = f"[{self._host}]" if ":" in self._host else self._host
+ return f"http://{host}:{self._generation_cut_port}"
+
@property
def engine_env(self) -> dict[str, str]:
"""The environment each engine's Dynamo worker needs to register."""
if not self._started:
raise RuntimeError("Managed Dynamo runtime has not been started")
# An OS-assigned system port makes each engine serve its own model-card
- # files to a frontend on another node.
+ # files (a custom chat template included) to a frontend on another node.
return {**self._manager_env, "DYN_SYSTEM_PORT": "0"}
def start(self) -> None:
@@ -179,22 +240,34 @@ def start(self) -> None:
)
http = (DEFAULT_DYNAMO_HTTP_PORT_RANGE_LOW, DEFAULT_DYNAMO_HTTP_PORT_RANGE_HIGH)
ports: list[int] = []
- for low, high in (control, control, control, http):
+ for low, high in (control, control, control, http, control):
ports.append(
_get_free_port_local(
low, high, max_retries=None, excluded_ports=set(ports)
)
)
- self._etcd_port, self._etcd_peer_port, self._nats_port, self._frontend_port = (
- ports
- )
+ (
+ self._etcd_port,
+ self._etcd_peer_port,
+ self._nats_port,
+ self._frontend_port,
+ self._generation_cut_port,
+ ) = ports
self._manager_env = {
"ETCD_ENDPOINTS": f"http://{self._host}:{self._etcd_port}",
"NATS_SERVER": f"nats://{self._host}:{self._nats_port}",
"DYN_NAMESPACE": self._namespace,
"DYN_DISCOVERY_BACKEND": "etcd",
+ # A field the frontend doesn't know fails the call with a 400 rather
+ # than vanishing; an inherited setting can't turn that off.
+ "DYN_IGNORE_OPENAI_FE_UNSUPPORTED_FIELDS": "0",
"DYN_REQUEST_PLANE": "tcp",
"DYN_EVENT_PLANE": "nats",
+ # NeMo-Gym sends media inline: raise the frontend's 45 MB HTTP body
+ # limit, and the 32 MiB request-plane limit above it (the engine
+ # request carries the same media plus token IDs).
+ "DYN_HTTP_BODY_LIMIT_MB": "512",
+ "DYN_TCP_MAX_MESSAGE_SIZE": str(1 << 30),
# A canary request would stall behind a refit pause; engine liveness
# is the Ray actor's, as for every other vLLM worker.
"DYN_HEALTH_CHECK_ENABLED": "false",
@@ -321,14 +394,26 @@ def _start_frontend(self) -> None:
namespace=self._namespace,
dynamo_cfg=self._dynamo_cfg,
)
+ # Dynamo's frontend, with NeMo RL's chat engine serving chat calls.
command = [
self._frontend_python,
"-m",
- "dynamo.frontend",
+ "nemo_rl.models.generation.dynamo.vllm.frontend",
*argv,
]
print(f" [Dynamo] launching frontend argv={command!r}", flush=True)
+ self._chat_dir = tempfile.mkdtemp(prefix="nemorl_dynamo_chat_")
frontend_env = self._service_env()
+ frontend_env[RENDER_WORKERS_ENV] = str(
+ self._dynamo_cfg.frontend_args.render_workers
+ )
+ frontend_env[CAPTURE_FILE_ENV] = os.path.join(self._chat_dir, "capture.pkl")
+ if self.serves_chat:
+ frontend_env[CHAT_SETTINGS_FILE_ENV] = os.path.join(
+ self._chat_dir, "chat_settings.json"
+ )
+ frontend_env[RAY_ADDRESS_ENV] = self._ray_address
+ frontend_env[GENERATION_CUT_PORT_ENV] = str(self._generation_cut_port)
dynamo_venv = os.path.dirname(os.path.dirname(self._frontend_python))
frontend_env["VIRTUAL_ENV"] = dynamo_venv
frontend_env["UV_PROJECT_ENVIRONMENT"] = dynamo_venv
@@ -336,6 +421,57 @@ def _start_frontend(self) -> None:
command, env=frontend_env, start_new_session=True
)
+ def publish_chat_capture(
+ self,
+ *,
+ dp_cfg: dict[str, Any],
+ staging_partition: str,
+ capture_media: bool,
+ generation_cut_control_token: str | None,
+ generation_cut_control_timeout_s: float | None,
+ ) -> None:
+ """Tell the frontend where token-capture prefixes are read, and who may cut.
+
+ Capture is set up after the engines register, so the chat engine reads
+ this on its first captured call. ``generation_cut_control_token`` is the
+ bearer token a generation cut must carry, None to refuse every cut;
+ ``generation_cut_control_timeout_s`` bounds one cut across the engines.
+ """
+ # Pickled: the data-plane config holds pydantic storage configs.
+ self._write_chat_file(
+ "capture.pkl",
+ pickle.dumps(
+ {
+ "data_plane": dp_cfg,
+ "staging_partition": staging_partition,
+ "capture_media": capture_media,
+ "generation_cut_control_token": generation_cut_control_token,
+ "generation_cut_control_timeout_s": (
+ generation_cut_control_timeout_s
+ ),
+ }
+ ),
+ )
+
+ def publish_chat_settings(self, settings: dict[str, Any]) -> None:
+ """Hand the frontend the chat settings an engine sent before it loaded.
+
+ ``settings`` is what the engine publishes at registration
+ (``chat_runtime_data``). The frontend builds its render workers from
+ them while the engines load, and registration confirms them.
+ """
+ self._write_chat_file("chat_settings.json", json.dumps(settings).encode())
+
+ def _write_chat_file(self, name: str, content: bytes) -> None:
+ if self._chat_dir is None:
+ raise RuntimeError("Managed Dynamo runtime has not been started")
+ with tempfile.NamedTemporaryFile(
+ "wb", dir=self._chat_dir, delete=False
+ ) as chat_file:
+ chat_file.write(content)
+ # Renamed into place, so the chat engine never reads a partial file.
+ os.replace(chat_file.name, os.path.join(self._chat_dir, name))
+
def wait_for_engines(
self, expected_engines: int, check_engines: Callable[[], None]
) -> None:
@@ -457,6 +593,9 @@ def shutdown(self) -> None:
if self._etcd_data_dir is not None:
shutil.rmtree(self._etcd_data_dir, ignore_errors=True)
self._etcd_data_dir = None
+ if self._chat_dir is not None:
+ shutil.rmtree(self._chat_dir, ignore_errors=True)
+ self._chat_dir = None
if self._nats_data_dir is not None:
shutil.rmtree(self._nats_data_dir, ignore_errors=True)
self._nats_data_dir = None
diff --git a/nemo_rl/models/generation/dynamo/vllm/__init__.py b/nemo_rl/models/generation/dynamo/vllm/__init__.py
new file mode 100644
index 00000000000..4fc25d0d3c9
--- /dev/null
+++ b/nemo_rl/models/generation/dynamo/vllm/__init__.py
@@ -0,0 +1,13 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/nemo_rl/models/generation/dynamo/vllm/adapter.py b/nemo_rl/models/generation/dynamo/vllm/adapter.py
new file mode 100644
index 00000000000..94970160419
--- /dev/null
+++ b/nemo_rl/models/generation/dynamo/vllm/adapter.py
@@ -0,0 +1,622 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Serve a NeMo RL vLLM worker's own AsyncLLM through Dynamo's backend SDK.
+
+The engine process holds no Dynamo-owned vLLM code. ``DynamoVllmAdapter`` is the
+vLLM backend of ``engine_adapter.DynamoEngineAdapter``, over the AsyncLLM that
+``VllmAsyncGenerationWorkerImpl`` owns, so refit, pause, sleep, quantization and
+speculative decoding are that worker's own.
+
+It serves two request kinds, both token ids in: NeMo RL's own rollouts, and
+NeMo-Gym's chat calls, whose prompt NeMo RL's chat engine in the frontend
+rendered, prepared and routed (``chat_engine``). For those the adapter runs the
+native server's steps after that (``prompt_and_capture``): ``start_capture``, the
+generation with the SamplingParams the native server would build, and
+``save_capture``'s one TQ write.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import dataclasses
+import json
+import logging
+import math
+import os
+import pickle
+import time
+from collections.abc import AsyncGenerator
+from typing import TYPE_CHECKING, Any, Optional
+
+import torch
+from dynamo._core import Context
+from dynamo.common.backend import (
+ EngineConfig,
+ GenerateChunk,
+ GenerateRequest,
+ LlmRegistration,
+)
+from dynamo.common.backend.publisher import ComponentSnapshot, KvEventSource, ZmqSource
+from dynamo.llm.exceptions import InvalidArgument
+from pydantic_core import to_jsonable_python
+from vllm.entrypoints.serve.utils.api_utils import get_max_tokens
+from vllm.inputs import TokensPrompt
+from vllm.v1.kv_cache_interface import KVCacheSpecKind
+from vllm.v1.metrics.loggers import StatLoggerBase
+
+from nemo_rl.models.generation.dynamo.config import DynamoCfg
+from nemo_rl.models.generation.dynamo.engine_adapter import (
+ DynamoEngineAdapter,
+ RequestHandler,
+ ServedRollout,
+)
+from nemo_rl.models.generation.dynamo.vllm.chat_engine import (
+ CALL_KEY,
+ GYM_CALL_ANNOTATION,
+ RUNTIME_DATA_KEY,
+ ForwardedCall,
+ GymEngineReply,
+)
+from nemo_rl.models.generation.dynamo.vllm.generation_cut import (
+ CUT_PREFIXES_KEY,
+ CUT_REPLY_MARGIN_S,
+ DEADLINE_KEY,
+ GENERATION_CUT_ANNOTATION,
+ INVENTORY_KEY,
+)
+from nemo_rl.models.generation.gym_call import GymCallResult
+from nemo_rl.models.generation.score_centering import (
+ extract_score_centering_samples,
+)
+from nemo_rl.models.generation.vllm.prompt_and_capture import (
+ PreparedPrompt,
+ abort_capture,
+ abort_request_capture,
+ save_capture,
+ start_capture,
+)
+from nemo_rl.models.generation.vllm.utils import (
+ chat_generation_tokens,
+ chat_routed_experts,
+)
+
+LOGGER = logging.getLogger(__name__)
+
+if TYPE_CHECKING:
+ from nemo_rl.models.generation.vllm.vllm_worker_async import (
+ VllmAsyncGenerationWorkerImpl,
+ )
+
+# vLLM's main-attention kinds, as Dynamo's own vLLM backend counts them
+# (dynamo/vllm/cache_info.py MAIN_ATTENTION_KV_CACHE_KINDS, which NeMo RL may
+# not import); test_main_attention_kinds_match_dynamos checks the copy.
+_MAIN_ATTENTION_KINDS = (
+ KVCacheSpecKind.FULL_ATTENTION,
+ KVCacheSpecKind.MLA_ATTENTION,
+ KVCacheSpecKind.SINK_FULL_ATTENTION,
+)
+
+
+def dynamo_kv_events_config(
+ cfg: DynamoCfg, *, prefix_caching: bool
+) -> Optional[dict[str, Any]]:
+ """The ``kv_events_config`` that feeds Dynamo's KV router.
+
+ None unless the frontend routes by KV and the engine caches prefixes: vLLM
+ publishes block events only for the blocks its prefix cache stores.
+ """
+ if cfg.frontend_args.router_mode != "kv" or not prefix_caching:
+ return None
+ return {
+ "enable_kv_cache_events": True,
+ "publisher": "zmq",
+ # A random abstract-namespace socket per engine (dynamo-integration.md).
+ "endpoint": f"ipc://@nemo-rl-kv-events-{os.urandom(8).hex()}",
+ }
+
+
+def chat_runtime_data(host: "VllmAsyncGenerationWorkerImpl") -> dict[str, Any]:
+ """What the frontend's chat engine renders with, published at registration.
+
+ The engine's own args (those a JSON payload can carry, which include every
+ one ``ModelConfig`` reads), so the frontend builds the same tokenizer,
+ template and processor; and all of ``policy.generation``, which the native
+ server's chat helpers read.
+ """
+ engine_args: dict[str, Any] = {}
+ for field in dataclasses.fields(host.llm_async_engine_args):
+ value = getattr(host.llm_async_engine_args, field.name)
+ if field.default is not dataclasses.MISSING:
+ default = field.default
+ elif field.default_factory is not dataclasses.MISSING:
+ default = field.default_factory()
+ else:
+ default = dataclasses.MISSING
+ try:
+ if value == default:
+ continue
+ json.dumps(value)
+ except (TypeError, ValueError):
+ continue
+ engine_args[field.name] = value
+ # The frontend renders chat calls, so it always loads the tokenizer.
+ engine_args["skip_tokenizer_init"] = False
+ return {
+ RUNTIME_DATA_KEY: {
+ "engine_args": engine_args,
+ # MasterConfig validates some blocks into pydantic models
+ # (sampler_replay); anything else that is not JSON fails here.
+ "generation": to_jsonable_python(host.cfg),
+ }
+ }
+
+
+class SnapshotStatLogger(StatLoggerBase):
+ """Push per-iteration KV usage into Dynamo's SnapshotPublisher (router load)."""
+
+ # Set by Dynamo after vLLM built the logger, so shared through the class.
+ publisher: Optional[Any] = None
+
+ def __init__(self, vllm_config: Any, engine_index: int = 0) -> None:
+ self.dp_rank = engine_index
+ # vLLM sizes the KV cache before it builds its stat loggers.
+ self.num_gpu_blocks = vllm_config.cache_config.num_gpu_blocks or 0
+
+ def record(self, scheduler_stats, iteration_stats, *args, **kwargs) -> None:
+ cls = type(self)
+ if scheduler_stats is None or cls.publisher is None:
+ return
+ total = max(1, self.num_gpu_blocks)
+ usage = scheduler_stats.kv_cache_usage
+ hit_rate = None
+ prefix = scheduler_stats.prefix_cache_stats
+ if prefix.queries > 0:
+ hit_rate = prefix.hits / prefix.queries
+ cls.publisher.publish(
+ self.dp_rank,
+ ComponentSnapshot(
+ kv_used_blocks=int(total * usage),
+ kv_total_blocks=total,
+ gpu_cache_usage=usage,
+ kv_cache_hit_rate=hit_rate,
+ dp_rank=self.dp_rank,
+ ),
+ )
+
+ def log_engine_initialized(self) -> None:
+ pass
+
+
+class DynamoVllmAdapter(DynamoEngineAdapter):
+ """The vLLM backend, over a running NeMo RL vLLM worker it does not own."""
+
+ def __init__(
+ self, host: "VllmAsyncGenerationWorkerImpl", *, leader_worker_idx: int
+ ):
+ super().__init__(leader_worker_idx=leader_worker_idx)
+ self._host = host
+ self._llm = host.llm
+ self._max_model_len = self._llm.vllm_config.model_config.max_model_len
+
+ def _handlers(self) -> dict[str, RequestHandler]:
+ return {
+ **super()._handlers(),
+ GYM_CALL_ANNOTATION: self._generate_gym,
+ GENERATION_CUT_ANNOTATION: self._generate_cut,
+ }
+
+ async def _generate_cut(
+ self, request: GenerateRequest, annotation: str, context: Context
+ ) -> AsyncGenerator[GenerateChunk, None]:
+ """Cut the calls of a checkpoint's inventory that this engine holds.
+
+ The frontend sends every engine the whole inventory
+ (:mod:`~nemo_rl.models.generation.dynamo.vllm.generation_cut`), so the
+ answer names only the calls cut here.
+ """
+ del annotation, context
+ # Deferred: Gym is optional outside framework-owned capture.
+ from nemo_gym._checkpoint.generation_cut import GenerationCutInventory
+
+ extra_args = request.get("extra_args") or {}
+ inventory = GenerationCutInventory.model_validate(extra_args[INVENTORY_KEY])
+ deadline = extra_args.get(DEADLINE_KEY)
+ if deadline is not None:
+ # The frontend's wall-clock deadline on this host's monotonic clock,
+ # less the time the answer needs to get back.
+ deadline = time.monotonic() + (deadline - time.time()) - CUT_REPLY_MARGIN_S
+ receipt = await self._host._prefix_recovery.run_cut(
+ inventory, deadline=deadline
+ )
+ yield {
+ "token_ids": [],
+ "index": 0,
+ "finish_reason": "stop",
+ "engine_data": {
+ CUT_PREFIXES_KEY: [
+ ack.model_dump(mode="json")
+ for ack in receipt.prefixes
+ if ack.disposition == "durable_prefix"
+ ]
+ },
+ }
+
+ async def serve_rollout(
+ self,
+ prompt_token_ids: list[int],
+ *,
+ greedy: bool,
+ stop_strings: Optional[list[str]],
+ request_id: str,
+ ) -> ServedRollout:
+ result, final_output = await self._host._serve_rollout(
+ TokensPrompt(prompt_token_ids=prompt_token_ids),
+ prompt_token_ids=prompt_token_ids,
+ greedy=greedy,
+ stop_strings=stop_strings,
+ request_id=request_id,
+ device=torch.device("cpu"),
+ )
+ if final_output is None:
+ return ServedRollout(result, finish_reason="length", usage=None)
+ return ServedRollout(
+ result,
+ finish_reason=final_output.outputs[0].finish_reason,
+ usage=_usage(final_output, len(result.token_ids)),
+ )
+
+ async def start(self, worker_id: int) -> EngineConfig:
+ del worker_id
+ vllm_config = self._llm.vllm_config
+ cache = vllm_config.cache_config
+ scheduler = vllm_config.scheduler_config
+ # The KV-event block size is the main-attention group's, which can
+ # differ from cache_config.block_size on hybrid models.
+ block_size = cache.block_size
+ for group in await self._llm.engine_core.call_utility_async(
+ "get_kv_cache_group_metadata"
+ ):
+ if group.get("kind") in _MAIN_ATTENTION_KINDS:
+ block_size = group["block_size"]
+ break
+ model = self._host.model_name
+ return EngineConfig(
+ model=model,
+ served_model_name=model,
+ runtime_data=chat_runtime_data(self._host),
+ # No token budget: the engine owns overflow (dynamo-integration.md).
+ llm=LlmRegistration(
+ context_length=self._max_model_len,
+ kv_cache_block_size=block_size,
+ total_kv_blocks=cache.num_gpu_blocks,
+ max_num_seqs=scheduler.max_num_seqs,
+ max_num_batched_tokens=scheduler.max_num_batched_tokens,
+ ),
+ )
+
+ async def _generate_gym(
+ self, request: GenerateRequest, annotation: str, context: Context
+ ) -> AsyncGenerator[GenerateChunk, None]:
+ # Deferred: chat_server imports vLLM's OpenAI server stack.
+ from nemo_rl.models.generation.vllm.chat_server import (
+ chat_request_from_render,
+ )
+
+ del annotation
+ wire: Optional[ForwardedCall] = (request.get("extra_args") or {}).get(CALL_KEY)
+ if wire is None:
+ raise InvalidArgument(
+ "NeMo-Gym calls reach NeMo RL's engines through NeMo RL's chat "
+ f"engine in the Dynamo frontend (extra_args.{CALL_KEY})."
+ )
+ host = self._host
+ plan = PreparedPrompt.from_json(
+ wire["prepared_prompt"],
+ prompt_token_ids=list(request["token_ids"]),
+ parse_admission=host._parse_capture_admission,
+ )
+ chat = chat_request_from_render(wire["request"])
+ try:
+ chunk = await self._serve_gym_call(chat, plan, wire, context)
+ except BaseException as error:
+ failure = await abort_capture(host, chat, error)
+ if failure is None:
+ raise
+ message = failure.message
+ if failure.code != failure.status:
+ # The frontend forwards only a status and a message.
+ message = f"{message} ({failure.code})"
+ chunk = _refusal(failure.status, message, acknowledge=failure.acknowledge)
+ if chunk is not None:
+ yield chunk
+
+ async def _serve_gym_call(
+ self, chat: Any, plan: PreparedPrompt, wire: ForwardedCall, context: Context
+ ) -> Optional[GenerateChunk]:
+ """One NeMo-Gym chat call, after the frontend prepared its prompt.
+
+ The native server's steps after the plan, in its order: the queue
+ check vLLM's serving runs first, :func:`start_capture` (a restore
+ from a checkpoint cut, and ``begin_call``, which stamps the weight
+ version), the generation, and :func:`save_capture`'s one TQ write.
+ Returns one chunk whose ``engine_data`` carries the output the
+ frontend builds the reply from and the call's :class:`GymCallResult`;
+ None for a captured call whose client went away.
+ """
+ host = self._host
+ # The worker's own engine client, as the native server uses it, with
+ # the queue limits vLLM's serving checks before the render.
+ engine_client = host._http_engine_client or self._llm
+ engine_client.check_admission(1, context.id())
+ engine_prompt: dict[str, Any]
+ media = wire["multimodal"]
+ if media is not None:
+ # The processed media, in the template's coordinates.
+ engine_prompt = {
+ **pickle.loads(base64.b64decode(media["inputs"])),
+ "prompt_token_ids": list(media["template_token_ids"]),
+ }
+ else:
+ engine_prompt = TokensPrompt(prompt_token_ids=plan.prompt_token_ids)
+ await start_capture(
+ host, chat, engine_prompt, plan, max_model_len=self._max_model_len
+ )
+ captured = chat in host._capture_registry
+ final_output = await self._generate(
+ chat, engine_prompt, wire, engine_client, context
+ )
+ if plan.admission is not None and context.is_stopped():
+ # Dynamo aborts the vLLM request on a disconnect but never cancels
+ # this generator: fail the capture, as natively, instead of staging it.
+ await abort_request_capture(host, chat, "client_disconnect")
+ return None
+ completion = final_output.outputs[0]
+ if completion.finish_reason == "error":
+ # As vLLM's chat server does (_raise_if_error): never stage a failed call.
+ raise RuntimeError(f"Generation failed for request {context.id()}")
+ output: dict[str, Any] = {
+ "text": completion.text,
+ "token_ids": list(completion.token_ids),
+ "finish_reason": completion.finish_reason,
+ "stop_reason": completion.stop_reason,
+ }
+ if captured:
+ result = await self._stage(chat, final_output)
+ else:
+ result = self._call_result(final_output)
+ if chat.logprobs:
+ output["logprobs"] = _wire_logprobs(completion.logprobs)
+ reply: GymEngineReply = {
+ "output": output,
+ "result": result.model_dump(exclude_none=True),
+ }
+ if chat._restored_prompt_token_ids:
+ # The frontend's reply puts the saved text back in front.
+ terminal = chat._restored_prefix_terminal
+ reply["restored"] = {
+ "prompt_token_ids": list(chat._restored_prompt_token_ids),
+ "token_ids": list(chat._restored_generation_token_ids),
+ "terminal": None
+ if terminal is None
+ else {
+ "prefix_token_count": terminal.prefix_token_count,
+ "reason": terminal.reason,
+ "finish_reason": terminal.finish_reason,
+ "stop_reason": terminal.stop_reason,
+ },
+ }
+ return {
+ "token_ids": output["token_ids"],
+ "index": 0,
+ "finish_reason": completion.finish_reason,
+ "completion_usage": _usage(final_output, len(completion.token_ids)),
+ "engine_data": reply,
+ }
+
+ async def _generate(
+ self,
+ chat: Any,
+ engine_prompt: Any,
+ wire: ForwardedCall,
+ engine_client: Any,
+ context: Context,
+ ) -> Any:
+ """Generate as the native server does, with the SamplingParams it builds."""
+ model_config = self._llm.model_config
+ default_sampling_params = model_config.get_diff_sampling_param()
+ override_max_tokens = (
+ default_sampling_params.get("max_tokens")
+ if model_config.generation_config not in ("auto", "vllm")
+ else getattr(model_config, "override_generation_config", {}).get(
+ "max_new_tokens"
+ )
+ )
+ max_tokens = get_max_tokens(
+ self._max_model_len,
+ chat.max_completion_tokens
+ if chat.max_completion_tokens is not None
+ else chat.max_tokens,
+ len(engine_prompt["prompt_token_ids"]),
+ default_sampling_params,
+ override_max_tokens,
+ truncate_prompt_tokens=chat.truncate_prompt_tokens,
+ )
+ params = chat.to_sampling_params(max_tokens, default_sampling_params)
+ if chat.ng_capture is not None:
+ # As the native request does: a cut needs the call's resolved budget.
+ self._host._prefix_recovery.on_sampling_params(chat, params)
+ final_output = None
+ async for output in engine_client.generate(
+ engine_prompt,
+ params,
+ context.id(),
+ reasoning_ended=wire["reasoning_ended"],
+ reasoning_parser_kwargs=wire["reasoning_parser_kwargs"],
+ ):
+ # Recorded before asking for the next output, which releases
+ # vLLM's finished state for this request.
+ self._host._prefix_recovery.observe(chat, output)
+ final_output = output
+ if final_output is None:
+ raise RuntimeError(f"No output received for request {context.id()}")
+ return final_output
+
+ def _routes(self, final_output: Any, *, encode_for_wire: bool) -> Any:
+ """The call's routes, as the native endpoint attaches them; None if off.
+
+ With router replay on, a call vLLM returned no routes for fails
+ (``chat_routed_experts`` raises), captured or not.
+ """
+ if not self._host._return_routed_experts_enabled():
+ return None
+ completion = final_output.outputs[0]
+ return chat_routed_experts(
+ final_output,
+ completion,
+ prompt_token_count=len(final_output.prompt_token_ids),
+ device=torch.device("cpu"),
+ logger=LOGGER,
+ routed_experts_dtype=self._host.routed_experts_dtype,
+ encode_for_wire=encode_for_wire,
+ )
+
+ def _call_result(self, final_output: Any) -> GymCallResult:
+ """An uncaptured call's tokens and routes, as the native message carries them."""
+ completion = final_output.outputs[0]
+ generation_log_probs = None
+ if completion.logprobs is not None:
+ _, generation_log_probs = chat_generation_tokens(completion)
+ for index, log_prob in enumerate(generation_log_probs):
+ if not math.isfinite(log_prob):
+ # Strict JSON has no NaN or inf (Dynamo sends null). The native
+ # server's JSONResponse fails this call with 500, which Gym retries.
+ raise RuntimeError(
+ f"Generation produced a non-finite log-prob ({log_prob}) at "
+ f"token {index} of {len(generation_log_probs)}"
+ )
+ return GymCallResult(
+ prompt_token_ids=list(final_output.prompt_token_ids),
+ generation_token_ids=list(completion.token_ids),
+ generation_log_probs=generation_log_probs,
+ routed_experts=self._routes(final_output, encode_for_wire=True),
+ )
+
+ async def _stage(self, chat: Any, final_output: Any) -> GymCallResult:
+ """Stage a captured call from this process, in one TQ write."""
+ # Deferred: only captured calls reach here, and capture already imports nemo_gym.
+ from nemo_gym.token_id_capture import NG_COMMIT_COORDS_FIELD
+
+ if self._host._capture_top_logprobs:
+ # Off the loop, as natively: the sampler top-k scales with the
+ # completion's length and K.
+ content = await asyncio.to_thread(self._response_to_stage, final_output)
+ else:
+ content = self._response_to_stage(final_output)
+ content = await save_capture(self._host, chat, content)
+ return GymCallResult(ng_commit_coords=content[NG_COMMIT_COORDS_FIELD])
+
+ def _response_to_stage(self, final_output: Any) -> dict[str, Any]:
+ """A captured call's response, with the fields the native endpoint stages."""
+ completion = final_output.outputs[0]
+ message: dict[str, Any] = {"generation_token_ids": list(completion.token_ids)}
+ if completion.logprobs is not None:
+ _, message["generation_log_probs"] = chat_generation_tokens(completion)
+ routes = self._routes(final_output, encode_for_wire=False)
+ if routes is not None:
+ message["routed_experts"] = routes
+ if self._host._capture_top_logprobs:
+ # Score centering stages the sampler's top-k with the call.
+ (
+ message["sampler_topk_token_ids"],
+ message["sampler_topk_logprobs"],
+ ) = extract_score_centering_samples(
+ completion, num_candidates=self._host._capture_top_logprobs
+ )
+ return {
+ "choices": [
+ {
+ "index": 0,
+ "finish_reason": completion.finish_reason,
+ "message": message,
+ }
+ ]
+ }
+
+ async def abort(self, context: Context) -> None:
+ await self._llm.abort(context.id())
+
+ async def kv_event_sources(self) -> list[KvEventSource]:
+ kv_events = self._llm.vllm_config.kv_events_config
+ if kv_events is None or not kv_events.enable_kv_cache_events:
+ return []
+ # A user's own kv_events_config can keep vLLM's wildcard default
+ # (tcp://*:5557): vLLM binds it, but a subscriber cannot connect to "*".
+ return [ZmqSource(endpoint=kv_events.endpoint.replace("*", "127.0.0.1"))]
+
+ def component_metrics_dp_ranks(self) -> list[int]:
+ return [0]
+
+ def attach_snapshot_publisher(self, publisher: Any) -> None:
+ SnapshotStatLogger.publisher = publisher
+
+ async def cleanup(self) -> None:
+ # The worker owns the AsyncLLM; its shutdown() tears the engine down.
+ SnapshotStatLogger.publisher = None
+
+
+def _wire_logprobs(logprobs: Any) -> Optional[list[dict[str, list[Any]]]]:
+ """Per-position logprobs as JSON, for the frontend's OpenAI ``logprobs``."""
+ if logprobs is None:
+ return None
+ return [
+ {
+ str(token_id): [entry.logprob, entry.rank, entry.decoded_token]
+ for token_id, entry in position.items()
+ }
+ for position in logprobs
+ ]
+
+
+def _refusal(status: int, message: str, *, acknowledge: bool = False) -> GenerateChunk:
+ """A Gym call the engine refused before it ran, answered in the chunk.
+
+ Dynamo hands an error the engine raises to the frontend's chat engine
+ without its status, so it could only answer 500; the chat engine answers
+ this with ``status``, as the native server does.
+ """
+ reply: GymEngineReply = {
+ "refusal": {
+ "status": int(status),
+ "message": message,
+ "acknowledge": acknowledge,
+ }
+ }
+ return {
+ "token_ids": [],
+ "index": 0,
+ "finish_reason": "stop",
+ "engine_data": reply,
+ }
+
+
+def _usage(output: Any, num_output_tokens: int) -> dict[str, Any]:
+ prompt_tokens = len(output.prompt_token_ids or ())
+ return {
+ "prompt_tokens": prompt_tokens,
+ "completion_tokens": num_output_tokens,
+ "total_tokens": prompt_tokens + num_output_tokens,
+ "prompt_tokens_details": {"cached_tokens": output.num_cached_tokens or 0},
+ }
diff --git a/nemo_rl/models/generation/dynamo/vllm/chat_engine.py b/nemo_rl/models/generation/dynamo/vllm/chat_engine.py
new file mode 100644
index 00000000000..2c65bc9b05e
--- /dev/null
+++ b/nemo_rl/models/generation/dynamo/vllm/chat_engine.py
@@ -0,0 +1,636 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""NeMo RL's chat engine inside Dynamo's frontend, for NeMo-Gym's calls.
+
+Dynamo's frontend hands every chat call to this engine (``chat_engine_factory``)
+instead of its Rust preprocessor, so NeMo-Gym's calls are rendered, spliced and
+parsed by the same code as NeMo RL's native vLLM server:
+
+1. the call's :class:`~nemo_rl.models.generation.gym_call.GymCallExtras` and
+ on-policy sampling are checked;
+2. a render-pool worker renders it (:mod:`render_pool`);
+3. here, the prompt is prepared as natively
+ (:func:`~nemo_rl.models.generation.vllm.prompt_and_capture.prepare_prompt`: the
+ token-capture prefix read from TQ, the splice and the clamp);
+4. the final token ids go through Dynamo's KV router to one engine, whose
+ adapter runs the capture and generation
+ (``adapter.DynamoVllmAdapter``);
+5. a render-pool worker builds the reply with vLLM's own parsers.
+
+The engine publishes its engine args and chat settings at registration
+(``EngineConfig.runtime_data``), so this engine renders exactly as it would.
+For a run that serves chat it also sends them before it loads, and the render
+workers are built from them meanwhile (:class:`ChatEngineFactory`).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import importlib
+import logging
+import os
+import threading
+import time
+import uuid
+from collections.abc import Coroutine
+from typing import Any, AsyncGenerator, Optional, TypedDict
+
+from dynamo.llm import PythonAsyncEngine
+from dynamo.llm.exceptions import HttpError, InvalidArgument
+
+from nemo_rl.models.generation.dynamo.engine_adapter import weight_free_model_path
+from nemo_rl.models.generation.dynamo.managed_runtime import (
+ RAY_ADDRESS_ENV,
+ RENDER_WORKERS_ENV,
+ read_chat_capture,
+ read_chat_settings,
+)
+from nemo_rl.models.generation.dynamo.vllm.render_pool import (
+ RenderPool,
+ RestoredReply,
+)
+from nemo_rl.models.generation.gym_call import (
+ GYM_RESULT_NVEXT_KEY,
+ GymCallExtras,
+ GymCallResult,
+ GymRequestFields,
+)
+
+LOGGER = logging.getLogger(__name__)
+
+# The key the engine publishes its chat settings under in runtime_data.
+RUNTIME_DATA_KEY = "nemo_rl_chat"
+# The annotation the adapter serves NeMo-Gym calls by; the ForwardedCall rides
+# in extra_args under CALL_KEY.
+GYM_CALL_ANNOTATION = "nemo_gym:rollout"
+CALL_KEY = "nemo_rl"
+
+
+class ForwardedMedia(TypedDict):
+ """A media call's render, in the template's coordinates."""
+
+ # The render's multimodal input, pickled then base64-encoded.
+ inputs: str
+ # The template's token ids, which the input's placeholders index.
+ template_token_ids: list[int]
+
+
+class ForwardedCall(TypedDict):
+ """What the chat engine sends the adapter in ``extra_args[CALL_KEY]``.
+
+ The call's ``PreparedPrompt.to_json()`` and what the engine needs that the
+ native worker has in process: the rendered request, its media, and what
+ vLLM's chat serving hands the engine besides the prompt.
+ """
+
+ prepared_prompt: dict[str, Any]
+ # The rendered chat request, without messages.
+ request: dict[str, Any]
+ # None for a text call, which sends its prompt once: as the final token ids.
+ multimodal: Optional[ForwardedMedia]
+ reasoning_ended: Optional[bool]
+ reasoning_parser_kwargs: Optional[dict[str, Any]]
+
+
+class GymCallRefusal(TypedDict):
+ """A call the engine refused before it ran, with the native status."""
+
+ status: int
+ message: str
+ # Whether Gym is told the capture call was refused (a native 400).
+ acknowledge: bool
+
+
+class GymEngineReply(TypedDict, total=False):
+ """The adapter's ``engine_data`` for a Gym call: an output, or a refusal."""
+
+ # A served call: the completion the reply is built from, and its
+ # GymCallResult as JSON.
+ output: dict[str, Any]
+ result: dict[str, Any]
+ # Only for a call resumed from a checkpoint cut.
+ restored: RestoredReply
+ refusal: GymCallRefusal
+
+
+# Fields of Dynamo's chat request that are not vLLM request fields
+# (chat_template_args is renamed to vLLM's chat_template_kwargs).
+_DYNAMO_ONLY_FIELDS = {"nvext", "chat_template_args", "unsupported_fields"}
+
+
+def _capture_refusal_ack(extras: GymCallExtras) -> str:
+ """Gym's acknowledgement that a capture call was refused before it ran.
+
+ The native route sends it as a header; Dynamo's frontend forwards only an
+ error's status and message, so it ends the message instead.
+ """
+ if extras.ng_capture is None or extras.ng_measure:
+ # Native's /tokenize never acknowledges a measure call.
+ return ""
+ # Deferred: Gym is installed wherever a call carries ng_capture.
+ from nemo_gym.token_id_capture.external_capture import (
+ dynamo_capture_refusal_suffix,
+ )
+
+ return dynamo_capture_refusal_suffix(extras.ng_capture["model_call_id"])
+
+
+# Free-form JSON the client wrote (a tool's parameters, a response_format
+# schema, processor and media kwargs) keeps its nulls, as natively: a schema's
+# `"default": null` is rendered into the prompt.
+_CLIENT_JSON_KEYS = frozenset(
+ {"parameters", "schema", "mm_processor_kwargs", "media_io_kwargs"}
+)
+
+
+def _drop_none(value: Any) -> Any:
+ """Drop the Nones Dynamo adds for unset struct fields, which vLLM rejects.
+
+ An image part's unset ``detail`` arrives as None, for example.
+ """
+ if isinstance(value, dict):
+ return {
+ k: v if k in _CLIENT_JSON_KEYS else _drop_none(v)
+ for k, v in value.items()
+ if v is not None
+ }
+ if isinstance(value, list):
+ return [_drop_none(v) for v in value]
+ return value
+
+
+class _CaptureSource:
+ """The read-only TQ client the token-capture prefix is read with.
+
+ Built on the first captured call, from what ``setup_token_capture`` wrote
+ for the frontend: capture starts after the engines register.
+ """
+
+ def __init__(self) -> None:
+ self._prefix_cache: Any = None
+ # Calls run resolve_prefix on several threads at once; connect once.
+ self._connect_lock = threading.Lock()
+
+ def resolve_prefix(self, serving_admission: Any) -> list[int]:
+ """The admission's prefix, read from TQ (blocking; run off the loop)."""
+ started = time.monotonic()
+ from nemo_rl.data_plane.tq_token_sink import resolve_admission_prefix
+
+ if self._prefix_cache is None:
+ with self._connect_lock:
+ if self._prefix_cache is None:
+ self._prefix_cache = self._connect()
+ LOGGER.info(
+ "The first token-capture prefix read imported and "
+ "connected to TQ in %.1fs",
+ time.monotonic() - started,
+ )
+ return resolve_admission_prefix(serving_admission, self._prefix_cache)
+
+ @staticmethod
+ def _connect() -> Any:
+ import ray
+
+ from nemo_rl.data_plane import build_data_plane_client
+ from nemo_rl.data_plane.tq_token_sink import ChainPrefixCache, TQTokenSource
+
+ capture = read_chat_capture()
+ if capture is None:
+ raise RuntimeError(
+ "a token-capture call reached the Dynamo frontend before token "
+ "capture was set up for it"
+ )
+ if not ray.is_initialized():
+ ray.init(address=os.environ[RAY_ADDRESS_ENV], log_to_driver=False)
+ client = build_data_plane_client(
+ capture["data_plane"], bootstrap=False, segment_size=0
+ )
+ cache = ChainPrefixCache()
+ cache.install(
+ TQTokenSource(
+ client,
+ staging_partition=capture["staging_partition"],
+ capture_media=capture["capture_media"],
+ )
+ )
+ return cache
+
+
+def _log_failed_start(task: asyncio.Task[None]) -> None:
+ """Log why the chat engine's start failed; the first call fails the same way."""
+ if not task.cancelled() and task.exception() is not None:
+ LOGGER.error(
+ "NeMo RL's chat engine could not start its render workers",
+ exc_info=task.exception(),
+ )
+
+
+async def _start_chat_workers(pool: RenderPool) -> None:
+ """Do what the first chat call would otherwise wait for.
+
+ The render workers start, and the steps a call runs are imported (they
+ import vLLM) in a thread, so the frontend's event loop keeps serving.
+ A call that arrives first waits for the workers, as without this.
+ """
+ await asyncio.gather(
+ pool.start(),
+ asyncio.to_thread(
+ importlib.import_module,
+ "nemo_rl.models.generation.vllm.prompt_and_capture",
+ ),
+ )
+
+
+class _StepTimes:
+ """How long each step of a chat call took, for the first call's log line."""
+
+ def __init__(self) -> None:
+ self._last = time.monotonic()
+ self._seconds: dict[str, float] = {}
+
+ def mark(self, step: str) -> None:
+ now = time.monotonic()
+ self._seconds[step] = now - self._last
+ self._last = now
+
+ def __str__(self) -> str:
+ return ", ".join(f"{step} {took:.1f}s" for step, took in self._seconds.items())
+
+
+class NemoRLChatEngine:
+ """Serves NeMo-Gym's chat calls for one model's engines."""
+
+ def __init__(
+ self,
+ *,
+ pool: RenderPool,
+ routed_engine: Any,
+ spec: dict[str, Any],
+ capture: _CaptureSource,
+ ):
+ self._pool = pool
+ self._routed_engine = routed_engine
+ self._generation = spec["generation"]
+ self._capture = capture
+ # The first call, and the first one served, each log once: what the
+ # frontend's start-up cost NeMo-Gym's first step.
+ self._first_call = True
+ self._first_served_call = True
+
+ async def generate(
+ self, request: dict[str, Any], context: Any = None
+ ) -> AsyncGenerator[dict[str, Any], None]:
+ # Deferred: prompt_and_capture imports vLLM.
+ from nemo_rl.models.generation.vllm.prompt_and_capture import error_reply
+
+ extras = GymCallExtras.from_annotations(
+ (request.get("nvext") or {}).get("annotations") or ()
+ )
+ if extras is None:
+ raise HttpError(
+ 400,
+ "NeMo RL's Dynamo frontend serves NeMo-Gym's calls (the "
+ "nemo_rl:gym: annotation) on chat; NeMo RL's own rollouts use "
+ "/v1/completions.",
+ )
+ try:
+ chunk = await self._serve(request, extras, context)
+ except HttpError:
+ raise
+ except Exception as error:
+ # The steps before the engine refuse a call with these, as
+ # natively; the frontend holds no capture state to fail.
+ failure = error_reply(error)
+ if failure is None:
+ raise
+ ack = _capture_refusal_ack(extras) if failure.acknowledge else ""
+ raise HttpError(failure.status, failure.message + ack) from error
+ yield chunk
+
+ async def _serve(
+ self, request: dict[str, Any], extras: GymCallExtras, context: Any
+ ) -> dict[str, Any]:
+ # Deferred: prompt_and_capture imports vLLM.
+ from nemo_rl.models.generation.vllm.prompt_and_capture import prepare_prompt
+ from nemo_rl.models.generation.vllm.utils import (
+ OffPolicySamplingError,
+ on_policy_sampling,
+ )
+
+ body = _drop_none(
+ {k: v for k, v in request.items() if k not in _DYNAMO_ONLY_FIELDS}
+ )
+ if request.get("chat_template_args"):
+ body["chat_template_kwargs"] = request["chat_template_args"]
+ if (body.get("n") or 1) != 1:
+ # Capture stages one choice per call, and the trainer replays one.
+ raise OffPolicySamplingError(
+ f"request n={body['n']} is not supported; NeMo-Gym calls take n=1"
+ )
+ if body.get("stream"):
+ # vLLM's delta outputs would leave this path only the last piece.
+ raise OffPolicySamplingError(
+ "request stream=true is not supported; NeMo-Gym calls do not stream"
+ )
+ body.update(
+ on_policy_sampling(
+ temperature=body.get("temperature"),
+ top_p=body.get("top_p"),
+ top_k=body.get("top_k"),
+ min_p=body.get("min_p"),
+ repetition_penalty=body.get("repetition_penalty"),
+ presence_penalty=body.get("presence_penalty"),
+ frequency_penalty=body.get("frequency_penalty"),
+ logit_bias=body.get("logit_bias"),
+ generation_config=self._generation,
+ )
+ )
+ request_fields: set[str] = set(GymRequestFields.model_fields)
+ body.update(extras.model_dump(include=request_fields))
+ first_call, self._first_call = self._first_call, False
+ steps = _StepTimes()
+ info = await self._pool.info()
+ steps.mark("render workers")
+ if first_call:
+ LOGGER.info("The first chat call waited for the %s", steps)
+ rendered = await self._pool.render(body, measure=extras.ng_measure)
+ steps.mark("render")
+ plan = await prepare_prompt(
+ rendered,
+ resolve_prefix=self._capture.resolve_prefix,
+ turn_end_token_ids=info["turn_end_token_ids"],
+ max_model_len=info["max_model_len"],
+ measure_only=extras.ng_measure,
+ )
+ steps.mark("prefix and splice")
+ if extras.ng_measure:
+ return self._chunk(
+ request,
+ GymCallResult(
+ prompt_token_count=len(plan.prompt_token_ids),
+ ng_context_measured=True,
+ ),
+ message={"role": "assistant", "content": ""},
+ finish_reason="stop",
+ )
+ media: Optional[ForwardedMedia] = None
+ if rendered.multimodal is not None:
+ media = ForwardedMedia(
+ inputs=base64.b64encode(rendered.multimodal).decode(),
+ template_token_ids=rendered.template_token_ids,
+ )
+ wire: ForwardedCall = {
+ "prepared_prompt": plan.to_json(),
+ "request": rendered.request,
+ "multimodal": media,
+ "reasoning_ended": rendered.reasoning_ended,
+ "reasoning_parser_kwargs": rendered.reasoning_parser_kwargs,
+ }
+ preprocessed = {
+ "model": request["model"],
+ "token_ids": plan.prompt_token_ids,
+ "stop_conditions": {"max_tokens": plan.max_tokens},
+ "sampling_options": {
+ "temperature": body["temperature"],
+ "top_p": body["top_p"],
+ },
+ "output_options": {},
+ "eos_token_ids": [],
+ "annotations": [GYM_CALL_ANNOTATION],
+ "extra_args": {CALL_KEY: wire},
+ }
+ engine_data = await self._call_engine(preprocessed, context)
+ steps.mark("engine")
+ if (refusal := engine_data.get("refusal")) is not None:
+ # The engine refused the call before it ran, with the native status.
+ ack = _capture_refusal_ack(extras) if refusal["acknowledge"] else ""
+ raise HttpError(refusal["status"], refusal["message"] + ack)
+ result = GymCallResult.model_validate(engine_data["result"])
+ reply_request = rendered.request
+ if result.ng_commit_coords is not None:
+ # A captured call's logprobs were staged with it; as natively, its
+ # reply carries none.
+ reply_request = {**reply_request, "logprobs": False, "top_logprobs": None}
+ response = await self._pool.respond(
+ reply_request,
+ plan.prompt_token_ids,
+ engine_data["output"],
+ engine_data.get("restored"),
+ )
+ steps.mark("reply")
+ if self._first_served_call:
+ self._first_served_call = False
+ LOGGER.info("The first chat call served took: %s", steps)
+ (choice,) = response["choices"]
+ return self._chunk(
+ request,
+ result,
+ message=choice["message"],
+ finish_reason=choice["finish_reason"],
+ logprobs=choice.get("logprobs"),
+ usage=response.get("usage"),
+ )
+
+ async def _call_engine(
+ self, preprocessed: dict[str, Any], context: Any
+ ) -> GymEngineReply:
+ engine_data: Optional[GymEngineReply] = None
+ stream = await self._routed_engine.generate(preprocessed, context=context)
+ try:
+ async for item in stream:
+ data = item.data()
+ if data and data.get("engine_data"):
+ engine_data = data["engine_data"]
+ except (ValueError, InvalidArgument) as error:
+ # Dynamo raises an engine's error as a ValueError, or from 1.7 the
+ # engine adapter's InvalidArgument as itself (not a ValueError), and
+ # would answer either with 400. The engine sends every status it means
+ # as a refusal, so anything raised is an engine failure: 500, as natively.
+ raise HttpError(500, str(error)) from error
+ if engine_data is None:
+ raise HttpError(500, "the engine returned no result for this call")
+ return engine_data
+
+ @staticmethod
+ def _chunk(
+ request: dict[str, Any],
+ result: GymCallResult,
+ *,
+ message: dict[str, Any],
+ finish_reason: str,
+ logprobs: Optional[dict[str, Any]] = None,
+ usage: Optional[dict[str, Any]] = None,
+ ) -> dict[str, Any]:
+ """The OpenAI chunk Dynamo's frontend aggregates into the reply."""
+ delta: dict[str, Any] = {
+ "role": "assistant",
+ "content": message.get("content"),
+ }
+ # vLLM names it "reasoning"; Dynamo's OpenAI types "reasoning_content".
+ reasoning = message.get("reasoning", message.get("reasoning_content"))
+ if reasoning is not None:
+ delta["reasoning_content"] = reasoning
+ if message.get("tool_calls"):
+ delta["tool_calls"] = [
+ {**tool_call, "index": index}
+ for index, tool_call in enumerate(message["tool_calls"])
+ ]
+ chunk: dict[str, Any] = {
+ "id": f"chatcmpl-{uuid.uuid4().hex}",
+ "object": "chat.completion.chunk",
+ "created": int(time.time()),
+ "model": request["model"],
+ "choices": [
+ {
+ "index": 0,
+ "delta": delta,
+ "finish_reason": finish_reason,
+ "logprobs": logprobs,
+ }
+ ],
+ "nvext": {GYM_RESULT_NVEXT_KEY: result.model_dump(exclude_none=True)},
+ }
+ if usage is not None:
+ chunk["usage"] = usage
+ return chunk
+
+
+class ChatEngineFactory:
+ """Dynamo's ``chat_engine_factory``: one chat engine per model's engines.
+
+ The render workers are built from an engine's chat settings, which reach
+ the frontend when the engine registers. For a run that serves chat the
+ driver also hands over the settings one engine sends before it loads
+ (``read_chat_settings``): :meth:`start_render_workers` starts the workers
+ at launch and builds them from those, and registration keeps them when the
+ engines register the same ``policy.generation``.
+ """
+
+ def __init__(self) -> None:
+ self._capture = _CaptureSource()
+ # The workers started at launch and the settings they were built from
+ # (None until those arrive), until a registration takes them.
+ self._early_pool: Optional[RenderPool] = None
+ self._early_spec: Optional[dict[str, Any]] = None
+ # Held here: the event loop keeps only a weak reference to its tasks.
+ self._tasks: list[asyncio.Task[None]] = []
+
+ def start_render_workers(self) -> None:
+ """Start the render workers before any engine registers.
+
+ Call it on the frontend's loop, in a run that serves chat.
+ """
+ self._early_pool = self._new_pool(start=True)
+ self._run(self._build_early_pool(self._early_pool))
+
+ def _run(self, work: Coroutine[Any, Any, None]) -> None:
+ task = asyncio.get_running_loop().create_task(work)
+ task.add_done_callback(_log_failed_start)
+ self._tasks.append(task)
+
+ def _new_pool(self, *, start: bool) -> RenderPool:
+ pool = RenderPool(workers=int(os.environ[RENDER_WORKERS_ENV]))
+ if start:
+ self._run(_start_chat_workers(pool))
+ return pool
+
+ async def _build_early_pool(self, pool: RenderPool) -> None:
+ """Build ``pool`` from the settings an engine sends before it loads."""
+ launched = time.monotonic()
+ while (runtime_data := read_chat_settings()) is None:
+ if self._early_pool is not pool:
+ return
+ await asyncio.sleep(0.2)
+ spec = runtime_data[RUNTIME_DATA_KEY]
+ # The directory an engine registers from; registration brings Dynamo's
+ # copy of the same files.
+ model_dir = await weight_free_model_path(spec["generation"]["model_name"])
+ if self._early_pool is not pool:
+ # The engines registered meanwhile, and their settings built it.
+ return
+ self._early_spec = spec
+ pool.build(spec, model_dir)
+ LOGGER.info(
+ "Building the render workers from the chat settings an engine sent "
+ "%.1fs after the frontend launched, before it loaded",
+ time.monotonic() - launched,
+ )
+
+ def _take_early_pool(
+ self, spec: dict[str, Any], model_dir: str
+ ) -> Optional[RenderPool]:
+ """The workers started at launch, built for ``spec``; None without them.
+
+ Workers built from settings sent early are kept when the engines
+ register the same ``generation`` (``policy.generation``), which every
+ engine derives its engine args from. The engine args are not compared:
+ vLLM fills in more of them while it builds the engine (scheduler
+ limits, resolved model paths), and no render uses those.
+ """
+ pool, self._early_pool = self._early_pool, None
+ early_spec, self._early_spec = self._early_spec, None
+ if pool is None:
+ return None
+ if early_spec is None:
+ # No settings came before registration: the workers have imported
+ # vLLM and build now.
+ pool.build(spec, model_dir)
+ return pool
+ early, registered = early_spec["generation"], spec["generation"]
+ changed = sorted(
+ key
+ for key in early.keys() | registered.keys()
+ if early.get(key) != registered.get(key)
+ )
+ if changed:
+ LOGGER.warning(
+ "The engines registered policy.generation settings other than "
+ "the ones one sent before it loaded (%s); restarting the render "
+ "workers",
+ ", ".join(changed),
+ )
+ pool.shutdown()
+ return None
+ LOGGER.info(
+ "The engines registered the settings the render workers were built "
+ "from; keeping the workers"
+ )
+ return pool
+
+ async def chat_engine_factory(
+ self, instance_id: Any, mdc: Any, routed_engine: Any
+ ) -> PythonAsyncEngine:
+ del instance_id
+ runtime_data = (mdc.runtime_config() or {}).get("runtime_data") or {}
+ spec = runtime_data.get(RUNTIME_DATA_KEY)
+ if spec is None:
+ raise RuntimeError(
+ f"model {mdc.name()!r} registered without NeMo RL's chat settings "
+ f"(runtime_data[{RUNTIME_DATA_KEY!r}]); only NeMo RL's engines are "
+ "served by this frontend"
+ )
+ pool = self._take_early_pool(spec, mdc.local_dir())
+ if pool is None:
+ # NeMo-Gym's calls need this flag, and the native worker starts its
+ # chat server on it. Without it no chat call comes, so nothing starts.
+ pool = self._new_pool(
+ start=bool(spec["generation"]["vllm_cfg"].get("expose_http_server"))
+ )
+ pool.build(spec, mdc.local_dir())
+ engine = NemoRLChatEngine(
+ pool=pool, routed_engine=routed_engine, spec=spec, capture=self._capture
+ )
+ return PythonAsyncEngine(engine.generate, asyncio.get_running_loop())
diff --git a/nemo_rl/models/generation/dynamo/vllm/frontend.py b/nemo_rl/models/generation/dynamo/vllm/frontend.py
new file mode 100644
index 00000000000..a6cbfd02d35
--- /dev/null
+++ b/nemo_rl/models/generation/dynamo/vllm/frontend.py
@@ -0,0 +1,75 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Run Dynamo's frontend with NeMo RL's chat engine serving chat calls.
+
+``python -m nemo_rl.models.generation.dynamo.vllm.frontend ``
+
+The arguments are ``dynamo.frontend``'s own. Its ``main`` builds one
+``EntrypointArgs``; this wraps that name to add NeMo RL's
+``chat_engine_factory``, so every chat call reaches
+:mod:`~nemo_rl.models.generation.dynamo.vllm.chat_engine` and never Dynamo's
+Rust preprocessor. ``/v1/completions`` (NeMo RL's own rollouts, token ids in)
+is unchanged. It wraps ``DistributedRuntime`` the same way, to serve NeMo-Gym's
+checkpoint generation cuts through the frontend's own runtime
+(:mod:`~nemo_rl.models.generation.dynamo.vllm.generation_cut`) and, in a run
+that serves chat, to start the render workers at launch.
+"""
+
+import os
+import sys
+
+
+def main() -> None:
+ import dynamo.frontend.main as frontend_main
+
+ from nemo_rl.models.generation.dynamo.managed_runtime import (
+ CHAT_SETTINGS_FILE_ENV,
+ )
+ from nemo_rl.models.generation.dynamo.vllm.chat_engine import ChatEngineFactory
+ from nemo_rl.models.generation.dynamo.vllm.generation_cut import (
+ start_generation_cut_server,
+ )
+
+ factory = ChatEngineFactory()
+ entrypoint_args = frontend_main.EntrypointArgs
+
+ def with_chat_engine(engine_type, **kwargs):
+ if "chat_engine_factory" in kwargs:
+ raise RuntimeError(
+ "NeMo RL's frontend serves chat with its own chat engine; remove "
+ "--dyn-chat-processor from dynamo_cfg.frontend_args.extra_cli_args"
+ )
+ return entrypoint_args(
+ engine_type, chat_engine_factory=factory.chat_engine_factory, **kwargs
+ )
+
+ distributed_runtime = frontend_main.DistributedRuntime
+
+ def on_the_frontends_loop(loop, *args, **kwargs):
+ runtime = distributed_runtime(loop, *args, **kwargs)
+ start_generation_cut_server(runtime, loop)
+ if CHAT_SETTINGS_FILE_ENV in os.environ:
+ # The driver names the file for a run that serves chat.
+ factory.start_render_workers()
+ return runtime
+
+ frontend_main.EntrypointArgs = with_chat_engine
+ frontend_main.DistributedRuntime = on_the_frontends_loop
+ sys.argv[0] = "dynamo.frontend"
+ frontend_main.main()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/nemo_rl/models/generation/dynamo/vllm/generation_cut.py b/nemo_rl/models/generation/dynamo/vllm/generation_cut.py
new file mode 100644
index 00000000000..35a4313be20
--- /dev/null
+++ b/nemo_rl/models/generation/dynamo/vllm/generation_cut.py
@@ -0,0 +1,205 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Checkpoint generation cuts for NeMo-Gym calls served through Dynamo.
+
+At a checkpoint NeMo-Gym posts the calls it has in flight to
+``/ng-control/v1/generation-cut`` beside the base URL they went to. Behind
+Dynamo that URL is the frontend, and the calls run in the engines. So the
+frontend process serves the route on a control port of its own and sends the
+cut to every engine over Dynamo's request plane. Each engine answers for the
+calls it cut, and the answers are merged into the one receipt Gym expects.
+
+The control port is there because a Dynamo frontend route can only be a
+synchronous GET (ai-dynamo/dynamo#18486). Once routes take POST and async
+handlers, this route can move onto the frontend's own port.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import threading
+import time
+from collections.abc import Iterable, Sequence
+from typing import Any
+
+from nemo_rl.models.generation.dynamo.managed_runtime import (
+ DYNAMO_COMPONENT,
+ DYNAMO_ENDPOINT,
+ GENERATION_CUT_PORT_ENV,
+ read_chat_capture,
+)
+
+LOGGER = logging.getLogger(__name__)
+
+# Marks an engine request as a generation cut; its inventory rides in extra_args.
+GENERATION_CUT_ANNOTATION = "nemo_rl:generation_cut"
+INVENTORY_KEY = "nemo_rl_generation_cut"
+# The calls an engine cut, in its answer's engine_data.
+CUT_PREFIXES_KEY = "prefixes"
+# The frontend's deadline for the cut, in extra_args: wall-clock seconds, since
+# an engine on another host does not share the frontend's monotonic clock.
+DEADLINE_KEY = "nemo_rl_generation_cut_deadline"
+# What an engine leaves of that deadline for its answer to reach the frontend.
+CUT_REPLY_MARGIN_S = 1.0
+
+
+def merge_cut_receipts(inventory: Any, engine_prefixes: Iterable[Sequence[Any]]) -> Any:
+ """Build Gym's receipt for ``inventory`` from what each engine cut.
+
+ An engine acknowledges only the calls it cut. A call no engine
+ acknowledged gets Gym's failure ack, so it regenerates after a restore.
+ """
+ from nemo_gym._checkpoint.generation_cut import (
+ GenerationCutPrefixAck,
+ GenerationCutReceipt,
+ )
+
+ cut: dict[str, Any] = {}
+ for prefixes in engine_prefixes:
+ for prefix in prefixes:
+ ack = GenerationCutPrefixAck.model_validate(prefix)
+ cut.setdefault(ack.ticket_id, ack)
+ return GenerationCutReceipt(
+ checkpoint_id=inventory.checkpoint_id,
+ cut_id=f"frontend-{inventory.inventory_digest}",
+ inventory_digest=inventory.inventory_digest,
+ inventory=inventory,
+ backend_snapshot_id=f"tq-{inventory.inventory_digest}",
+ prefixes=tuple(
+ cut.get(prefix.ticket_id) or GenerationCutPrefixAck.failure(prefix)
+ for prefix in inventory.active_prefixes
+ ),
+ )
+
+
+class EngineCutFanOut:
+ """Cuts a checkpoint's calls on every engine: what the cut route runs."""
+
+ def __init__(self, runtime: Any, loop: asyncio.AbstractEventLoop) -> None:
+ """Open the request-plane client. Call it on ``loop``, the frontend's.
+
+ The client is opened now, at startup, so that by the first checkpoint
+ it has discovered every engine.
+ """
+ # The frontend's loop, which the runtime's request-plane client runs on.
+ self._loop = loop
+ path = f"{os.environ['DYN_NAMESPACE']}.{DYNAMO_COMPONENT}.{DYNAMO_ENDPOINT}"
+ # Dynamo returns a future here, not a coroutine.
+ self._client = asyncio.ensure_future(runtime.endpoint(path).client())
+
+ @property
+ def control_token(self) -> str | None:
+ """The bearer token a cut needs; None until capture is set up with cuts on."""
+ capture = read_chat_capture()
+ return None if capture is None else capture["generation_cut_control_token"]
+
+ async def run_cut(self, inventory: Any) -> Any:
+ """Cut ``inventory``'s calls. Safe to await from the control server's loop."""
+ # One deadline, from the cut's arrival, for this wait and every engine:
+ # an engine that started its own later would answer after this gave up.
+ timeout_s = (read_chat_capture() or {}).get("generation_cut_control_timeout_s")
+ deadline = None if timeout_s is None else time.time() + timeout_s
+ return await asyncio.wrap_future(
+ asyncio.run_coroutine_threadsafe(
+ self._cut_on_every_engine(inventory, deadline), self._loop
+ )
+ )
+
+ async def _cut_on_every_engine(self, inventory: Any, deadline: float | None) -> Any:
+ client = await self._client
+ request = {
+ # Dynamo's request type requires the field: without it the request
+ # never reaches the engine and the caller is left waiting.
+ "token_ids": [],
+ "annotations": [GENERATION_CUT_ANNOTATION],
+ "extra_args": {
+ INVENTORY_KEY: inventory.model_dump(mode="json"),
+ DEADLINE_KEY: deadline,
+ },
+ }
+ cuts = {
+ asyncio.ensure_future(
+ self._cut_on_engine(client, request, instance_id)
+ ): instance_id
+ for instance_id in client.instance_ids()
+ }
+ engine_prefixes = []
+ if cuts:
+ done, late = await asyncio.wait(
+ cuts,
+ timeout=None if deadline is None else max(0.0, deadline - time.time()),
+ )
+ for cut in late:
+ cut.cancel()
+ if late:
+ LOGGER.warning(
+ "generation cut %s: engines %s did not answer in time; calls "
+ "they hold regenerate after a restore",
+ inventory.checkpoint_id,
+ sorted(cuts[cut] for cut in late),
+ )
+ engine_prefixes = [cut.result() for cut in done]
+ return merge_cut_receipts(inventory, engine_prefixes)
+
+ @staticmethod
+ async def _cut_on_engine(
+ client: Any, request: dict[str, Any], instance_id: int
+ ) -> Sequence[Any]:
+ prefixes: Sequence[Any] = ()
+ try:
+ stream = await client.direct(request, instance_id)
+ async for item in stream:
+ data = item.data()
+ if data and data.get("engine_data"):
+ prefixes = data["engine_data"][CUT_PREFIXES_KEY]
+ # One engine's failure must cost only the calls it holds, whatever it is.
+ except Exception: # noqa: BLE001
+ LOGGER.warning(
+ "generation cut failed on engine %s; calls it holds regenerate "
+ "after a restore",
+ instance_id,
+ exc_info=True,
+ )
+ return prefixes
+
+
+def start_generation_cut_server(runtime: Any, loop: asyncio.AbstractEventLoop) -> None:
+ """Serve Gym's generation-cut route on this frontend process's control port."""
+ # Deferred: only the frontend process serves the route.
+ import uvicorn
+ from fastapi import FastAPI
+
+ from nemo_rl.models.generation.vllm.generation_cut_route import (
+ register_generation_cut_route,
+ )
+
+ fan_out = EngineCutFanOut(runtime, loop)
+ app = FastAPI()
+ register_generation_cut_route(app, lambda: fan_out)
+ server = uvicorn.Server(
+ uvicorn.Config(
+ app,
+ host="0.0.0.0",
+ port=int(os.environ[GENERATION_CUT_PORT_ENV]),
+ log_level="warning",
+ )
+ )
+ # A thread of its own, as each engine's HTTP server has: on the frontend's
+ # loop uvicorn would take over the frontend's signal handlers.
+ threading.Thread(
+ target=server.run, name="nrl-generation-cut-server", daemon=True
+ ).start()
diff --git a/nemo_rl/models/generation/dynamo/vllm/render_pool.py b/nemo_rl/models/generation/dynamo/vllm/render_pool.py
new file mode 100644
index 00000000000..3bd2c775978
--- /dev/null
+++ b/nemo_rl/models/generation/dynamo/vllm/render_pool.py
@@ -0,0 +1,531 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Render and answer NeMo-Gym chat calls in worker processes of Dynamo's frontend.
+
+Rendering (template, tokenizer, the checkpoint's HF processor) and building the
+reply (detokenized text through vLLM's reasoning and tool parsers) are
+CPU-bound Python; on the frontend's one event loop they would serialize every
+call in flight. Each worker process builds the same vLLM objects NeMo RL's
+native chat server uses, from the engine's own engine args, and runs:
+
+- ``render``: the native server's render step (``_render_chat_call``: the
+ render, the capture decision, the history render the splice cuts against);
+- ``respond``: vLLM's own non-streaming response builder over the engine's
+ output, so the reply is the one the native server builds.
+
+Jobs queue FIFO in the pool; nothing renders on the event loop.
+
+A worker imports vLLM before it has the settings it is built from, so the
+frontend can start the workers at launch, before any engine has registered.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import importlib
+import json
+import logging
+import multiprocessing
+import os
+import pickle
+import shutil
+import tempfile
+import time
+import uuid
+from concurrent.futures import ProcessPoolExecutor
+from concurrent.futures.process import BrokenProcessPool
+from dataclasses import dataclass
+from typing import Any, Callable, Optional, TypedDict
+
+LOGGER = logging.getLogger(__name__)
+
+# Set in each worker process by _init_worker.
+_WORKER: Optional["_Worker"] = None
+
+
+class RestoredReply(TypedDict):
+ """What the engine resumed a call from, after a checkpoint cut it.
+
+ The engine swaps the frontend's prompt for the saved rows and generates
+ only the rest, so the reply needs these to read as the whole call's.
+ """
+
+ # The prompt the engine ran: the capture prefix, then the saved rows.
+ prompt_token_ids: list[int]
+ # The saved rows' generated tokens, which the reply's text and usage count.
+ token_ids: list[int]
+ # The saved prefix's finish, when it had already ended the call:
+ # ``_RestoredPrefixTerminal``'s fields without the prompt.
+ terminal: Optional[dict[str, Any]]
+
+
+@dataclass(frozen=True)
+class RenderedCall:
+ """What the frontend needs from a render to plan, route and serve a call.
+
+ A :class:`~nemo_rl.models.generation.vllm.prompt_and_capture.RenderedPrompt`.
+ """
+
+ template_token_ids: list[int]
+ # The render up to the last assistant turn, when a prefix will be spliced.
+ history_token_ids: Optional[list[int]]
+ # The request's own prefix (Gym's echo) after the capture decision.
+ required_prefix_token_ids: Optional[list[int]]
+ # Capture admissions: what is stored, and how the prompt is served.
+ admission: Any
+ serving_admission: Any
+ verify_retained_media: bool
+ # The rendered engine input's multimodal part (pickled), or None for text.
+ multimodal: Optional[bytes]
+ # The chat request as rendered (parsers may adjust it), without messages:
+ # the engine builds its SamplingParams from it, as the native server does.
+ request: dict[str, Any]
+ # The output-token budget the request asked for, before the clamp.
+ request_max_tokens: Optional[int]
+ # What vLLM's chat serving hands the engine besides the prompt.
+ reasoning_ended: Optional[bool]
+ reasoning_parser_kwargs: Optional[dict[str, Any]]
+
+
+def _import_vllm() -> None:
+ """Import everything a worker builds and serves with."""
+ import vllm.platforms
+
+ # A frontend host may have no accelerator; vLLM only renders here, as in
+ # Dynamo's own frontend. vLLM's explicit CPU target selects its CPU
+ # platform. An accelerator host keeps its own, which validates the
+ # engine's args (quantization, for one) as the engine does.
+ if vllm.platforms.resolve_current_platform_cls_qualname().endswith(
+ ".UnspecifiedPlatform"
+ ):
+ os.environ["VLLM_TARGET_DEVICE"] = "cpu"
+ # The chat server imports the rest of vLLM that a worker uses.
+ importlib.import_module("nemo_rl.models.generation.vllm.chat_server")
+
+
+class _Worker:
+ """The vLLM objects one worker process renders and responds with."""
+
+ def __init__(self, spec: dict[str, Any], model_dir: str):
+ started = time.monotonic()
+ _import_vllm()
+
+ from types import SimpleNamespace
+
+ from vllm.config import CacheConfig, LoadConfig, VllmConfig
+ from vllm.engine.arg_utils import AsyncEngineArgs
+ from vllm.entrypoints.openai.models.protocol import BaseModelPath
+ from vllm.entrypoints.openai.models.serving import OpenAIServingModels
+ from vllm.renderers.registry import renderer_from_config
+ from vllm.v1.engine.input_processor import InputProcessor
+
+ from nemo_rl.models.generation.openai_server_utils import turn_end_token_ids
+ from nemo_rl.models.generation.vllm import chat_server
+ from nemo_rl.models.generation.vllm.utils import model_stop_token_ids
+ from nemo_rl.models.generation.vllm.video_utils import (
+ register_vllm_video_loader,
+ )
+
+ cfg = spec["generation"]
+ chat_server.register_parser_plugins(cfg)
+ # This worker decodes the call's videos; the engine runs its pixels.
+ register_vllm_video_loader(cfg)
+ # The engine's own args, with the weight-free snapshot Dynamo fetched.
+ # Sleep mode is the engine's memory management, and vLLM's CPU platform
+ # refuses it.
+ args = AsyncEngineArgs(
+ **{
+ **spec["engine_args"],
+ "model": model_dir,
+ "tokenizer": model_dir,
+ "enable_sleep_mode": False,
+ }
+ )
+ self.model_config = args.create_model_config()
+ vllm_config = VllmConfig(
+ model_config=self.model_config,
+ load_config=LoadConfig(load_format="dummy"),
+ cache_config=CacheConfig(),
+ )
+ renderer = renderer_from_config(vllm_config)
+ self.tokenizer = renderer.tokenizer
+ serving_chat_kwargs, default_chat_template_kwargs = (
+ chat_server.serving_chat_settings(cfg)
+ )
+ self.online_renderer = chat_server.FrontendRenderer(
+ model_config=self.model_config,
+ renderer=renderer,
+ request_logger=None,
+ chat_template=serving_chat_kwargs["chat_template"],
+ chat_template_content_format=serving_chat_kwargs[
+ "chat_template_content_format"
+ ],
+ enable_auto_tools=serving_chat_kwargs["enable_auto_tools"],
+ tool_parser=serving_chat_kwargs.get("tool_parser"),
+ reasoning_parser=serving_chat_kwargs.get("reasoning_parser"),
+ default_chat_template_kwargs=default_chat_template_kwargs,
+ )
+ # As vLLM's API server does at startup: compiles the chat template and
+ # warms the multimodal processor, so the first render does neither.
+ self.online_renderer.warmup()
+ # vLLM's chat serving builds replies only here; it never generates.
+ engine_client = SimpleNamespace(
+ model_config=self.model_config,
+ renderer=renderer,
+ vllm_config=vllm_config,
+ input_processor=InputProcessor(vllm_config),
+ io_processor=None,
+ errored=False,
+ )
+ models = OpenAIServingModels(
+ engine_client=engine_client,
+ base_model_paths=[
+ BaseModelPath(
+ name=self.model_config.served_model_name, model_path=model_dir
+ )
+ ],
+ lora_modules=None,
+ )
+ self.serving_chat = chat_server.FrontendServingChat(
+ **{
+ **serving_chat_kwargs,
+ "engine_client": engine_client,
+ "models": models,
+ "online_renderer": self.online_renderer,
+ "return_tokens_as_token_ids": True,
+ }
+ )
+ self.loop = asyncio.new_event_loop()
+ # Every id a turn can end on: the splice in the frontend cuts by them.
+ self.turn_end_token_ids = sorted(
+ turn_end_token_ids(
+ model_stop_token_ids(self.model_config, cfg), self.tokenizer
+ )
+ )
+ self.build_seconds = time.monotonic() - started
+
+ def _parser(self, chat: Any) -> Any:
+ parser_cls = self.serving_chat.parser_cls
+ if parser_cls is None:
+ return None
+ return parser_cls(
+ self.tokenizer,
+ chat.tools,
+ chat_template_kwargs=self.serving_chat._effective_chat_template_kwargs(
+ chat
+ ),
+ model_config=self.model_config,
+ )
+
+ def render(self, body: dict[str, Any], *, measure: bool = False) -> RenderedCall:
+ from nemo_rl.models.generation.vllm.chat_server import (
+ NeMoRLChatCompletionRequest,
+ NeMoRLMeasureChatRequest,
+ )
+ from nemo_rl.models.generation.vllm.prompt_and_capture import set_max_tokens
+
+ request_cls = (
+ NeMoRLMeasureChatRequest if measure else NeMoRLChatCompletionRequest
+ )
+ chat = request_cls.model_validate(body)
+ admission = None
+ if chat.ng_capture is not None:
+ # Deferred: only captured calls need it, and capture already imports nemo_gym.
+ from nemo_gym.token_id_capture.staging.records import CaptureAdmission
+
+ admission = CaptureAdmission.model_validate(chat.ng_capture)
+ rendered = self.loop.run_until_complete(
+ self.online_renderer.render_for_frontend(chat, admission=admission)
+ )
+ # The serving reads these back from the request; restore the budget
+ # the render lowered for vLLM's pre-tokenization length check.
+ if rendered.request_max_tokens is not None:
+ set_max_tokens(chat, rendered.request_max_tokens)
+ parser = self._parser(chat)
+ reasoning_parser_kwargs = None
+ if parser is not None and parser.reasoning_parser is not None:
+ reasoning_parser_kwargs = {
+ "chat_template_kwargs": self.serving_chat._engine_chat_template_kwargs(
+ self.serving_chat._effective_chat_template_kwargs(chat)
+ )
+ }
+ multimodal = None
+ if rendered.engine_prompt.get("mm_kwargs"):
+ multimodal = pickle.dumps(
+ {
+ key: value
+ for key, value in rendered.engine_prompt.items()
+ if key != "prompt_token_ids"
+ },
+ protocol=pickle.HIGHEST_PROTOCOL,
+ )
+ return RenderedCall(
+ template_token_ids=list(rendered.engine_prompt["prompt_token_ids"]),
+ history_token_ids=rendered.history_token_ids,
+ required_prefix_token_ids=rendered.required_prefix_token_ids,
+ admission=rendered.admission,
+ serving_admission=rendered.serving_admission,
+ verify_retained_media=rendered.verify_retained_media,
+ multimodal=multimodal,
+ request=chat.model_dump(mode="json", exclude={"messages"}),
+ request_max_tokens=rendered.request_max_tokens,
+ reasoning_ended=rendered.reasoning_ended,
+ reasoning_parser_kwargs=reasoning_parser_kwargs,
+ )
+
+ def respond(
+ self,
+ request: dict[str, Any],
+ prompt_token_ids: list[int],
+ output: dict[str, Any],
+ restored: Optional[RestoredReply] = None,
+ ) -> dict[str, Any]:
+ from vllm.entrypoints.generate.base.protocol import RequestResponseMetadata
+ from vllm.entrypoints.openai.chat_completion.protocol import (
+ ChatCompletionResponse,
+ )
+ from vllm.logprobs import Logprob
+ from vllm.outputs import CompletionOutput, RequestOutput
+
+ from nemo_rl.models.generation.prefix_recovery.restore import (
+ _RestoredPrefixTerminal,
+ )
+ from nemo_rl.models.generation.vllm.chat_server import (
+ chat_request_from_render,
+ )
+
+ chat = chat_request_from_render(request)
+ if restored is not None:
+ # The engine resumed this call from the tokens a checkpoint cut
+ # saved: it ran this prompt and generated only the rest. The
+ # serving chat's reply fixes read these fields, as natively.
+ prompt_token_ids = restored["prompt_token_ids"]
+ chat._restored_prompt_token_ids = tuple(prompt_token_ids)
+ chat._restored_generation_token_ids = tuple(restored["token_ids"])
+ # The engine lowered its own copy's min_tokens, never this one.
+ chat._original_min_tokens = chat.min_tokens
+ if restored["terminal"] is not None:
+ chat._restored_prefix_terminal = _RestoredPrefixTerminal(
+ prompt_token_ids=tuple(prompt_token_ids), **restored["terminal"]
+ )
+ logprobs = None
+ if output.get("logprobs") is not None:
+ logprobs = [
+ {
+ int(token_id): Logprob(
+ logprob=entry[0], rank=entry[1], decoded_token=entry[2]
+ )
+ for token_id, entry in position.items()
+ }
+ for position in output["logprobs"]
+ ]
+ request_output = RequestOutput(
+ request_id=f"chatcmpl-{uuid.uuid4().hex}",
+ prompt=None,
+ prompt_token_ids=prompt_token_ids,
+ prompt_logprobs=None,
+ outputs=[
+ CompletionOutput(
+ index=0,
+ text=output["text"],
+ token_ids=output["token_ids"],
+ cumulative_logprob=None,
+ logprobs=logprobs,
+ finish_reason=output["finish_reason"],
+ stop_reason=output.get("stop_reason"),
+ )
+ ],
+ finished=True,
+ )
+
+ async def results():
+ yield request_output
+
+ response = self.loop.run_until_complete(
+ self.serving_chat.chat_completion_full_generator(
+ chat,
+ results(),
+ request_output.request_id,
+ request["model"],
+ [],
+ self.tokenizer,
+ RequestResponseMetadata(request_id=request_output.request_id),
+ parser=self._parser(chat),
+ )
+ )
+ if not isinstance(response, ChatCompletionResponse):
+ raise ValueError(f"vLLM could not build the reply: {response}")
+ return response.model_dump(mode="json", exclude_none=True)
+
+
+def _wait_for_settings(path: str) -> tuple[dict[str, Any], str]:
+ """The settings :meth:`RenderPool.build` writes, once it has."""
+ frontend = multiprocessing.parent_process()
+ while not os.path.exists(path):
+ # None will come once the pool is shut down or the frontend is gone.
+ if not os.path.isdir(os.path.dirname(path)) or (
+ frontend is not None and not frontend.is_alive()
+ ):
+ raise SystemExit("the render pool went away before it had settings")
+ time.sleep(0.1)
+ with open(path) as settings_file:
+ settings = json.load(settings_file)
+ return settings["spec"], settings["model_dir"]
+
+
+def _init_worker(settings_path: str) -> None:
+ global _WORKER
+ # Before the wait, so that only the build is left once the settings arrive.
+ _import_vllm()
+ _WORKER = _Worker(*_wait_for_settings(settings_path))
+
+
+def _render(body: dict[str, Any], measure: bool) -> RenderedCall:
+ assert _WORKER is not None
+ return _WORKER.render(body, measure=measure)
+
+
+def _respond(
+ request: dict[str, Any],
+ prompt_token_ids: list[int],
+ output: dict[str, Any],
+ restored: Optional[RestoredReply],
+) -> dict[str, Any]:
+ assert _WORKER is not None
+ return _WORKER.respond(request, prompt_token_ids, output, restored)
+
+
+def _info() -> dict[str, Any]:
+ assert _WORKER is not None
+ return {
+ "turn_end_token_ids": _WORKER.turn_end_token_ids,
+ "max_model_len": _WORKER.model_config.max_model_len,
+ "build_seconds": _WORKER.build_seconds,
+ }
+
+
+class RenderPool:
+ """``workers`` processes that render and answer chat calls, FIFO.
+
+ A worker imports vLLM, then waits for the settings :meth:`build` gives the
+ pool, so :meth:`start` can come first. :meth:`start` starts them all;
+ without it the executor starts one for each job submitted while none is
+ idle, so the first calls wait for them.
+ """
+
+ def __init__(self, *, workers: int):
+ self._workers = workers
+ # Where build() writes the settings; a restarted pool's workers read
+ # them here too.
+ self._settings_dir = tempfile.mkdtemp(prefix="nemorl_render_pool_")
+ self._settings_path = os.path.join(self._settings_dir, "settings.json")
+ self._built_at: Optional[float] = None
+ self._executor = self._start()
+ self._info: Optional[dict[str, Any]] = None
+
+ def build(self, spec: dict[str, Any], model_dir: str) -> None:
+ """Give the workers what they are built from.
+
+ ``spec`` is the engine's chat settings (``chat_runtime_data``), and
+ ``model_dir`` a local directory with the model's config and tokenizer
+ files.
+ """
+ if self._built_at is not None:
+ raise RuntimeError("the render pool already has its settings")
+ with tempfile.NamedTemporaryFile(
+ "w", dir=self._settings_dir, delete=False
+ ) as settings_file:
+ json.dump({"spec": spec, "model_dir": model_dir}, settings_file)
+ # Renamed into place, so a worker never reads a partial file.
+ os.replace(settings_file.name, self._settings_path)
+ self._built_at = time.monotonic()
+
+ def _start(self) -> ProcessPoolExecutor:
+ return ProcessPoolExecutor(
+ max_workers=self._workers,
+ # Fork would copy the frontend's Rust runtime and event loop.
+ mp_context=multiprocessing.get_context("spawn"),
+ initializer=_init_worker,
+ initargs=(self._settings_path,),
+ )
+
+ async def _submit(self, fn: Callable[..., Any], *args: Any) -> Any:
+ executor = self._executor
+ try:
+ return await asyncio.wrap_future(executor.submit(fn, *args))
+ except BrokenProcessPool as error:
+ if self._info is None:
+ # No worker ever served a job: most often _init_worker failed (a
+ # config error, which a restart cannot fix), so do not restart.
+ raise RuntimeError(
+ "the render pool's workers died before serving a call; see "
+ "this log for an 'Exception in initializer' traceback or a "
+ "kill (e.g. OOM)"
+ ) from error
+ if self._executor is executor:
+ # A worker died (e.g. OOM-killed). The calls in flight fail;
+ # later calls get a fresh pool instead of this broken one.
+ LOGGER.error("A render-pool worker died; restarting the pool")
+ executor.shutdown(wait=False, cancel_futures=True)
+ self._executor = self._start()
+ raise
+
+ async def start(self) -> None:
+ """Start every worker now: one job each, submitted while none is idle.
+
+ Returns when the first is ready, which takes :meth:`build`'s settings.
+ """
+ started = time.monotonic()
+ infos = await asyncio.gather(
+ *(self._submit(_info) for _ in range(self._workers))
+ )
+ self._info = infos[0]
+ ready = time.monotonic()
+ assert self._built_at is not None
+ LOGGER.info(
+ "The render pool's first of %d workers is ready %.1fs after they "
+ "started and %.1fs after their settings arrived; it took %.1fs to "
+ "build",
+ self._workers,
+ ready - started,
+ ready - self._built_at,
+ self._info["build_seconds"],
+ )
+
+ async def info(self) -> dict[str, Any]:
+ """The model's turn-end ids and context length, from a started worker."""
+ if self._info is None:
+ self._info = await self._submit(_info)
+ return self._info
+
+ async def render(
+ self, body: dict[str, Any], *, measure: bool = False
+ ) -> RenderedCall:
+ """Render a call; a measure call is counted past the context, as natively."""
+ return await self._submit(_render, body, measure)
+
+ async def respond(
+ self,
+ request: dict[str, Any],
+ prompt_token_ids: list[int],
+ output: dict[str, Any],
+ restored: Optional[RestoredReply] = None,
+ ) -> dict[str, Any]:
+ """Build the reply; ``restored`` is the engine's ``RestoredReply``, if any."""
+ return await self._submit(_respond, request, prompt_token_ids, output, restored)
+
+ def shutdown(self) -> None:
+ self._executor.shutdown(wait=False, cancel_futures=True)
+ shutil.rmtree(self._settings_dir, ignore_errors=True)
diff --git a/nemo_rl/models/generation/dynamo/vllm_adapter.py b/nemo_rl/models/generation/dynamo/vllm_adapter.py
deleted file mode 100644
index 9c55df7d666..00000000000
--- a/nemo_rl/models/generation/dynamo/vllm_adapter.py
+++ /dev/null
@@ -1,203 +0,0 @@
-# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Serve a NeMo RL vLLM worker's own AsyncLLM through Dynamo's backend SDK.
-
-The engine process holds no Dynamo-owned vLLM code. ``DynamoVllmAdapter`` is the
-vLLM backend of ``engine_adapter.DynamoEngineAdapter``, over the AsyncLLM that
-``VllmAsyncGenerationWorkerImpl`` owns, so refit, pause, sleep, quantization and
-speculative decoding are that worker's own.
-"""
-
-from __future__ import annotations
-
-import os
-from typing import TYPE_CHECKING, Any, Optional
-
-import torch
-from dynamo._core import Context
-from dynamo.common.backend import EngineConfig, LlmRegistration
-from dynamo.common.backend.publisher import ComponentSnapshot, KvEventSource, ZmqSource
-from vllm.inputs import TokensPrompt
-from vllm.v1.kv_cache_interface import KVCacheSpecKind
-from vllm.v1.metrics.loggers import StatLoggerBase
-
-from nemo_rl.models.generation.dynamo.config import DynamoCfg
-from nemo_rl.models.generation.dynamo.engine_adapter import (
- DynamoEngineAdapter,
- ServedRollout,
-)
-
-if TYPE_CHECKING:
- from nemo_rl.models.generation.vllm.vllm_worker_async import (
- VllmAsyncGenerationWorkerImpl,
- )
-
-# vLLM's main-attention kinds, as Dynamo's own vLLM backend counts them
-# (dynamo/vllm/cache_info.py MAIN_ATTENTION_KV_CACHE_KINDS, which NeMo RL may
-# not import); test_main_attention_kinds_match_dynamos checks the copy.
-_MAIN_ATTENTION_KINDS = (
- KVCacheSpecKind.FULL_ATTENTION,
- KVCacheSpecKind.MLA_ATTENTION,
- KVCacheSpecKind.SINK_FULL_ATTENTION,
-)
-
-
-def dynamo_kv_events_config(
- cfg: DynamoCfg, *, prefix_caching: bool
-) -> Optional[dict[str, Any]]:
- """The ``kv_events_config`` that feeds Dynamo's KV router.
-
- None unless the frontend routes by KV and the engine caches prefixes: vLLM
- publishes block events only for the blocks its prefix cache stores.
- """
- if cfg.frontend_args.router_mode != "kv" or not prefix_caching:
- return None
- return {
- "enable_kv_cache_events": True,
- "publisher": "zmq",
- # A random abstract-namespace socket per engine (dynamo-integration.md).
- "endpoint": f"ipc://@nemo-rl-kv-events-{os.urandom(8).hex()}",
- }
-
-
-class SnapshotStatLogger(StatLoggerBase):
- """Push per-iteration KV usage into Dynamo's SnapshotPublisher (router load)."""
-
- # Set by Dynamo after vLLM built the logger, so shared through the class.
- publisher: Optional[Any] = None
-
- def __init__(self, vllm_config: Any, engine_index: int = 0) -> None:
- self.dp_rank = engine_index
- # vLLM sizes the KV cache before it builds its stat loggers.
- self.num_gpu_blocks = vllm_config.cache_config.num_gpu_blocks or 0
-
- def record(self, scheduler_stats, iteration_stats, *args, **kwargs) -> None:
- cls = type(self)
- if scheduler_stats is None or cls.publisher is None:
- return
- total = max(1, self.num_gpu_blocks)
- usage = scheduler_stats.kv_cache_usage
- hit_rate = None
- prefix = scheduler_stats.prefix_cache_stats
- if prefix.queries > 0:
- hit_rate = prefix.hits / prefix.queries
- cls.publisher.publish(
- self.dp_rank,
- ComponentSnapshot(
- kv_used_blocks=int(total * usage),
- kv_total_blocks=total,
- gpu_cache_usage=usage,
- kv_cache_hit_rate=hit_rate,
- dp_rank=self.dp_rank,
- ),
- )
-
- def log_engine_initialized(self) -> None:
- pass
-
-
-class DynamoVllmAdapter(DynamoEngineAdapter):
- """The vLLM backend, over a running NeMo RL vLLM worker it does not own."""
-
- def __init__(
- self, host: "VllmAsyncGenerationWorkerImpl", *, leader_worker_idx: int
- ):
- super().__init__(leader_worker_idx=leader_worker_idx)
- self._host = host
- self._llm = host.llm
- self._max_model_len = self._llm.vllm_config.model_config.max_model_len
-
- async def serve_rollout(
- self,
- prompt_token_ids: list[int],
- *,
- greedy: bool,
- stop_strings: Optional[list[str]],
- request_id: str,
- ) -> ServedRollout:
- result, final_output = await self._host._serve_rollout(
- TokensPrompt(prompt_token_ids=prompt_token_ids),
- prompt_token_ids=prompt_token_ids,
- greedy=greedy,
- stop_strings=stop_strings,
- request_id=request_id,
- device=torch.device("cpu"),
- )
- if final_output is None:
- return ServedRollout(result, finish_reason="length", usage=None)
- return ServedRollout(
- result,
- finish_reason=final_output.outputs[0].finish_reason,
- usage=_usage(final_output, len(result.token_ids)),
- )
-
- async def start(self, worker_id: int) -> EngineConfig:
- del worker_id
- vllm_config = self._llm.vllm_config
- cache = vllm_config.cache_config
- scheduler = vllm_config.scheduler_config
- # The KV-event block size is the main-attention group's, which can
- # differ from cache_config.block_size on hybrid models.
- block_size = cache.block_size
- for group in await self._llm.engine_core.call_utility_async(
- "get_kv_cache_group_metadata"
- ):
- if group.get("kind") in _MAIN_ATTENTION_KINDS:
- block_size = group["block_size"]
- break
- model = self._host.model_name
- return EngineConfig(
- model=model,
- served_model_name=model,
- # No token budget: the engine owns overflow (dynamo-integration.md).
- llm=LlmRegistration(
- context_length=self._max_model_len,
- kv_cache_block_size=block_size,
- total_kv_blocks=cache.num_gpu_blocks,
- max_num_seqs=scheduler.max_num_seqs,
- max_num_batched_tokens=scheduler.max_num_batched_tokens,
- ),
- )
-
- async def abort(self, context: Context) -> None:
- await self._llm.abort(context.id())
-
- async def kv_event_sources(self) -> list[KvEventSource]:
- kv_events = self._llm.vllm_config.kv_events_config
- if kv_events is None or not kv_events.enable_kv_cache_events:
- return []
- # A user's own kv_events_config can keep vLLM's wildcard default
- # (tcp://*:5557): vLLM binds it, but a subscriber cannot connect to "*".
- return [ZmqSource(endpoint=kv_events.endpoint.replace("*", "127.0.0.1"))]
-
- def component_metrics_dp_ranks(self) -> list[int]:
- return [0]
-
- def attach_snapshot_publisher(self, publisher: Any) -> None:
- SnapshotStatLogger.publisher = publisher
-
- async def cleanup(self) -> None:
- # The worker owns the AsyncLLM; its shutdown() tears the engine down.
- SnapshotStatLogger.publisher = None
-
-
-def _usage(output: Any, num_output_tokens: int) -> dict[str, Any]:
- prompt_tokens = len(output.prompt_token_ids or ())
- return {
- "prompt_tokens": prompt_tokens,
- "completion_tokens": num_output_tokens,
- "total_tokens": prompt_tokens + num_output_tokens,
- "prompt_tokens_details": {"cached_tokens": output.num_cached_tokens or 0},
- }
diff --git a/nemo_rl/models/generation/gym_call.py b/nemo_rl/models/generation/gym_call.py
new file mode 100644
index 00000000000..87d933d4c9f
--- /dev/null
+++ b/nemo_rl/models/generation/gym_call.py
@@ -0,0 +1,87 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""The extra fields NeMo RL and NeMo-Gym exchange on a chat call, defined once.
+
+The native vLLM server's request classes inherit :class:`GymRequestFields`, and
+it answers with :class:`GymCallResult`'s fields on the message. Dynamo's
+frontend carries only OpenAI fields plus ``nvext``, so through it NeMo-Gym's
+Dynamo dialect packs :class:`GymCallExtras` (those request fields, plus the
+measure flag) into one ``nemo_rl:gym:`` annotation. NeMo RL's chat engine
+hands the request fields to the same render and returns the result as
+``nvext["nemo_rl"]``. A field added to :class:`GymRequestFields` reaches both
+paths.
+"""
+
+import json
+from collections.abc import Iterable
+from typing import Any, Optional
+
+from pydantic import BaseModel, ConfigDict
+
+# The annotation NeMo-Gym's Dynamo dialect carries the extras in.
+GYM_EXTRAS_ANNOTATION_PREFIX = "nemo_rl:gym:"
+# The response nvext key the call result rides in through Dynamo.
+GYM_RESULT_NVEXT_KEY = "nemo_rl"
+
+
+class GymRequestFields(BaseModel):
+ """The request fields both paths read; the native request classes inherit them."""
+
+ # The token-capture admission (capture_key, call ids, prev_len, mode, ...).
+ ng_capture: Optional[dict[str, Any]] = None
+ # Without capture: the last echoed assistant turn's prompt + generation ids.
+ required_prefix_token_ids: Optional[list[int]] = None
+
+
+class GymCallExtras(GymRequestFields):
+ """What a NeMo-Gym chat call adds to an OpenAI request through Dynamo."""
+
+ # A field Gym sends that NeMo RL does not know is refused, not ignored.
+ model_config = ConfigDict(extra="forbid")
+
+ # Return only the prompt's length (Gym's /context/measure); no generation.
+ # Natively Gym calls /tokenize for this instead.
+ ng_measure: bool = False
+
+ @classmethod
+ def from_annotations(cls, annotations: Iterable[str]) -> "GymCallExtras | None":
+ """The extras in ``nvext.annotations``; None when the call carries none."""
+ for annotation in annotations or ():
+ if annotation.startswith(GYM_EXTRAS_ANNOTATION_PREFIX):
+ return cls.model_validate(
+ json.loads(annotation[len(GYM_EXTRAS_ANNOTATION_PREFIX) :])
+ )
+ return None
+
+
+class GymCallResult(BaseModel):
+ """What a NeMo-Gym chat call returns beyond the OpenAI response.
+
+ The native server puts these on ``choices[0].message`` (token fields,
+ ``routed_experts``) or the response (``ng_commit_coords``); field names are
+ the native ones, so Gym lifts them without renaming.
+ """
+
+ # A generation call without token capture.
+ prompt_token_ids: Optional[list[int]] = None
+ generation_token_ids: Optional[list[int]] = None
+ generation_log_probs: Optional[list[float]] = None
+ # Router replay: the wire-encoded routes, as the native message carries them.
+ routed_experts: Optional[Any] = None
+ # A captured call: where its record was staged.
+ ng_commit_coords: Optional[dict[str, Any]] = None
+ # A measure-only call.
+ prompt_token_count: Optional[int] = None
+ ng_context_measured: Optional[bool] = None
diff --git a/nemo_rl/models/generation/openai_server_utils.py b/nemo_rl/models/generation/openai_server_utils.py
index f01cd1f6091..0f520f75eb4 100644
--- a/nemo_rl/models/generation/openai_server_utils.py
+++ b/nemo_rl/models/generation/openai_server_utils.py
@@ -27,6 +27,19 @@
from typing import Any
+def turn_end_token_ids(
+ stop_token_ids: Collection[int], tokenizer: Any
+) -> frozenset[int]:
+ """Every id a turn can end on, for :func:`splice_prefix_tokens`.
+
+ vLLM stops on the generation config's EOS as well as the configured stops,
+ and the chat template closes turns with the tokenizer's EOS, so a turn can
+ end on any of them. ``stop_token_ids`` is vLLM's
+ :func:`nemo_rl.models.generation.vllm.utils.model_stop_token_ids`.
+ """
+ return frozenset(stop_token_ids) | {tokenizer.eos_token_id}
+
+
@dataclass(frozen=True)
class PrefixSplice:
"""Exact token splice and its source coordinates for multimodal ranges."""
@@ -42,7 +55,7 @@ def replace_prefix_tokens(
template_prefix_token_ids: list[int],
template_token_ids: list[int],
*,
- eos_token_id: int | Collection[int] | None = None,
+ eos_token_id: int | Collection[int],
) -> list[int]:
"""Replace a rendered history with the exact previously generated tokens.
@@ -64,7 +77,7 @@ def splice_prefix_tokens(
model_prefix_token_ids: list[int],
template_prefix_token_ids: list[int],
template_token_ids: list[int],
- eos_token_id: int | Collection[int] | None = None,
+ eos_token_id: int | Collection[int],
) -> PrefixSplice:
"""This is a subroutine used inside the OpenAI-compatible Chat Completion server.
@@ -124,21 +137,18 @@ def splice_prefix_tokens(
resumes from the template after that EOS:
output => [11,12,13,40,41,220,17,2,21,22,40,41]
- ``eos_token_id`` overrides ``tokenizer.eos_token_id``; with it, ``tokenizer``
- may be ``None`` (callers that only hold token ids, e.g. the Megatron prompt
- preparer) and the failure message skips the detokenized reprs. It may also
- be a collection of ids for models that declare several EOS tokens (the
- Megatron chat endpoint ships the model's full EOS set): any of them counts
- toward the boundary, and when the model's prefix ends on one of them that
- exact id is kept in place of the template's, so the spliced prompt still
- starts with the model's tokens verbatim.
+ ``eos_token_id`` is every id a turn can end on (:func:`turn_end_token_ids`
+ for vLLM); the tokenizer's own EOS alone misses a turn that ended on another
+ stop (gpt-oss ``<|call|>``). ``tokenizer`` may be ``None`` (callers that only
+ hold token ids, e.g. the Megatron prompt preparer); the failure message then
+ skips the detokenized reprs. Any id in the set counts toward the boundary,
+ and when the model's prefix ends on one of them that exact id is kept in
+ place of the template's, so the spliced prompt still starts with the
+ model's tokens verbatim.
"""
if not model_prefix_token_ids:
return PrefixSplice(template_token_ids, 0, 0)
- if eos_token_id is None:
- eos_token_id = tokenizer.eos_token_id
- assert eos_token_id is not None, "Tokenizer must have an EOS token ID"
eos_token_ids = (
frozenset([eos_token_id])
if isinstance(eos_token_id, int)
diff --git a/nemo_rl/models/generation/prefix_recovery/recovery.py b/nemo_rl/models/generation/prefix_recovery/recovery.py
index 7f568f1aa97..cdae3228103 100644
--- a/nemo_rl/models/generation/prefix_recovery/recovery.py
+++ b/nemo_rl/models/generation/prefix_recovery/recovery.py
@@ -37,7 +37,6 @@
from nemo_rl.models.generation.prefix_recovery.parts import PrefixRecoveryParts
from nemo_rl.models.generation.prefix_recovery.restore import (
_generation_cut_blockers,
- restore_reply_text,
)
from nemo_rl.models.generation.prefix_recovery.state import (
_CompletedCaptureState,
@@ -201,21 +200,6 @@ def observe(self, request: Any, request_output: Any) -> None:
stop_reason=stop_reason,
)
- def restore_reply_text(
- self, request: Any, request_output: Any, *, tokenizer: Any
- ) -> None:
- """Prepend a resumed call's restored text to its final output."""
- state = self._registry.get(request)
- if state is None or not state.resumed_generation_token_ids:
- return
- restore_reply_text(
- request,
- request_output,
- prefix_token_ids=state.resumed_generation_token_ids,
- engine_prompt_token_ids=state.prompt_token_ids,
- tokenizer=tokenizer,
- )
-
def retire(self, staging_keys: Sequence[str]) -> None:
"""Queue an ended call's chunk rows for the next snapshot to delete.
@@ -282,11 +266,17 @@ def shutdown(self) -> None:
# ---- A checkpoint cut -----------------------------------------------
- async def run_cut(self, inventory: Any) -> Any:
- """Cut every call Gym's inventory names, off the serving loop."""
- # Measured from arrival so time queued behind a stale cut counts.
- timeout_s = self._parts.control_timeout_s
- deadline = None if timeout_s is None else time.monotonic() + timeout_s
+ async def run_cut(self, inventory: Any, *, deadline: float | None = None) -> Any:
+ """Cut every call Gym's inventory names, off the serving loop.
+
+ ``deadline`` is a ``time.monotonic()`` bound from a caller that waits
+ on one of its own (the Dynamo frontend). Without it the cut gets
+ ``control_timeout_s`` from its arrival here.
+ """
+ if deadline is None:
+ # Measured from arrival so time queued behind a stale cut counts.
+ timeout_s = self._parts.control_timeout_s
+ deadline = None if timeout_s is None else time.monotonic() + timeout_s
return await asyncio.get_running_loop().run_in_executor(
self._control_executor,
functools.partial(self.checkpoint_cut, inventory, deadline=deadline),
@@ -880,11 +870,6 @@ def on_sampling_params(self, request: Any, sampling_params: Any) -> None:
def observe(self, request: Any, request_output: Any) -> None:
pass
- def restore_reply_text(
- self, request: Any, request_output: Any, *, tokenizer: Any
- ) -> None:
- pass
-
def remember_completed(
self,
model_call_id: str,
@@ -910,5 +895,5 @@ def counts(self) -> dict[str, int]:
def shutdown(self) -> None:
pass
- async def run_cut(self, inventory: Any) -> Any:
+ async def run_cut(self, inventory: Any, *, deadline: float | None = None) -> Any:
raise RuntimeError("generation-prefix cuts are disabled on this worker")
diff --git a/nemo_rl/models/generation/prefix_recovery/restore.py b/nemo_rl/models/generation/prefix_recovery/restore.py
index c8c42635309..805d5723d51 100644
--- a/nemo_rl/models/generation/prefix_recovery/restore.py
+++ b/nemo_rl/models/generation/prefix_recovery/restore.py
@@ -344,7 +344,7 @@ def _full_generator_signature(method: Callable[..., Any]) -> inspect.Signature:
raise RuntimeError(
"vLLM's OpenAIServingChat.chat_completion_full_generator has no "
f"{missing} parameter, which restored generation prefixes rewrite; "
- "update NeMoRLOpenAIServingChatMixin.chat_completion_full_generator"
+ "update ResumedCallReplyMixin.chat_completion_full_generator"
)
return signature
diff --git a/nemo_rl/models/generation/trtllm/trtllm_http_server.py b/nemo_rl/models/generation/trtllm/trtllm_http_server.py
index 035481c1c3a..bc2e6de3c04 100644
--- a/nemo_rl/models/generation/trtllm/trtllm_http_server.py
+++ b/nemo_rl/models/generation/trtllm/trtllm_http_server.py
@@ -168,7 +168,7 @@ def _sync() -> list[int]:
model_prefix_token_ids=required_prefix_ids,
template_prefix_token_ids=template_prefix_ids,
template_token_ids=prompt_token_ids,
- eos_token_id=eos_token_ids or None,
+ eos_token_id=eos_token_ids or tokenizer.eos_token_id,
)
return await asyncio.to_thread(_sync)
diff --git a/nemo_rl/models/generation/vllm/chat_server.py b/nemo_rl/models/generation/vllm/chat_server.py
new file mode 100644
index 00000000000..4a5fe0cc135
--- /dev/null
+++ b/nemo_rl/models/generation/vllm/chat_server.py
@@ -0,0 +1,1008 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""NeMo RL's OpenAI chat and tokenize server over a vLLM engine.
+
+The vLLM async worker builds it for its HTTP server. The classes live here, not
+inside the worker, so another host of the same chat path (Dynamo's frontend)
+can build the same renderer and chat serving. ``host`` is the worker: it owns
+token capture. This module imports vLLM's OpenAI server stack; import it only
+where vLLM is installed.
+"""
+
+import asyncio
+import copy
+import logging
+from copy import deepcopy
+from dataclasses import dataclass, replace
+from logging import Filter as LoggingFilter
+from logging import LogRecord
+from typing import Any, Literal, Optional, Union
+
+import torch
+from fastapi import FastAPI, Request
+from fastapi.responses import JSONResponse, StreamingResponse
+from pydantic import PrivateAttr
+from vllm.entrypoints.openai.chat_completion.protocol import (
+ ChatCompletionRequest,
+ ChatCompletionResponse,
+)
+from vllm.entrypoints.openai.chat_completion.serving import OpenAIServingChat
+from vllm.entrypoints.openai.models.protocol import BaseModelPath
+from vllm.entrypoints.openai.models.serving import OpenAIServingModels
+
+# vLLM 0.29 moved this out of the openai package (vllm-project/vllm#54492).
+from vllm.entrypoints.serve.engine.protocol import ErrorResponse
+from vllm.entrypoints.serve.tokenize.protocol import (
+ TokenizeChatRequest,
+ TokenizeCompletionRequest,
+ TokenizeResponse,
+)
+from vllm.entrypoints.serve.tokenize.serving import ServingTokenization
+from vllm.entrypoints.serve.utils.api_utils import with_cancellation
+from vllm.exceptions import VLLMClientError, VLLMValidationError
+from vllm.reasoning.abs_reasoning_parsers import ReasoningParserManager
+from vllm.renderers.chat_utils.chat_template import load_chat_template
+from vllm.renderers.hf import HfRenderer
+from vllm.renderers.online_renderer import OnlineRenderer
+from vllm.sampling_params import StructuredOutputsParams
+from vllm.tool_parsers.abstract_tool_parser import ToolParserManager
+from vllm.v1.engine.async_llm import logger as vllm_async_llm_logger
+
+from nemo_rl.models.generation.capture_context import resolve_capture_input
+from nemo_rl.models.generation.gym_call import GymCallResult, GymRequestFields
+from nemo_rl.models.generation.openai_server_utils import turn_end_token_ids
+from nemo_rl.models.generation.prefix_recovery.restore import (
+ _full_generator_signature,
+ _RestoredPrefixTerminal,
+ restore_reply_text,
+ restore_reply_usage,
+ wrap_parser_for_restore,
+)
+from nemo_rl.models.generation.vllm.captured_chat_logprobs import (
+ CapturedChatLogprobsMixin,
+)
+from nemo_rl.models.generation.vllm.generation_cut_route import (
+ register_generation_cut_route,
+)
+from nemo_rl.models.generation.vllm.prompt_and_capture import (
+ ErrorReply,
+ abort_capture,
+ abort_request_capture,
+ error_reply,
+ prepare_prompt,
+ save_capture,
+ set_max_tokens,
+ start_capture,
+)
+from nemo_rl.models.generation.vllm.utils import (
+ REASONING_ENDED_ARG,
+ attach_routed_experts_to_chat_response_choices,
+ attach_sampler_topk_logprobs_to_chat_response_choices,
+ attach_token_information_to_chat_response_choices,
+ model_dump_chat_response_with_dynamic_message_fields,
+ model_stop_token_ids,
+ on_policy_sampling,
+)
+
+LOGGER = logging.getLogger(__name__)
+
+
+class NeMoRLOpenAIChatRequestMixin:
+ # Declared by the request models this mixes into.
+ required_prefix_token_ids: Optional[list[int]]
+ messages: Any
+
+ def model_post_init(self, context):
+ # NeMo-Gym specific processing. This is just how NeMo-Gym returns the extra token information.
+ if self.required_prefix_token_ids is None:
+ for message in reversed(self.messages):
+ if "prompt_token_ids" in message:
+ self.required_prefix_token_ids = (
+ message["prompt_token_ids"] + message["generation_token_ids"]
+ )
+ break
+
+ return super().model_post_init(context)
+
+
+# This MRO is necessary i.e. NeMoRLOpenAIChatRequestMixin > ChatCompletionRequest
+class NeMoRLChatCompletionRequest(
+ NeMoRLOpenAIChatRequestMixin, GymRequestFields, ChatCompletionRequest
+):
+ # Set when the call resumes from the tokens a checkpoint cut saved.
+ _restored_prefix_terminal: _RestoredPrefixTerminal | None = PrivateAttr(
+ default=None
+ )
+ _restored_generation_token_ids: tuple[int, ...] = PrivateAttr(default=())
+ # The engine prompt of a restored call: the capture prefix, then the
+ # saved rows; empty otherwise.
+ _restored_prompt_token_ids: tuple[int, ...] = PrivateAttr(default=())
+ # The client's min_tokens before it is reduced by a restored prefix;
+ # replaying the prefix's stop handling needs the original window.
+ _original_min_tokens: int | None = PrivateAttr(default=None)
+ # reasoning_ended from the template render; None without a reasoning parser.
+ _template_reasoning_ended: bool | None = PrivateAttr(default=None)
+
+
+class NeMoRLMeasureChatRequest(NeMoRLChatCompletionRequest):
+ """A chat call Gym measures, counted as ``/tokenize`` counts it.
+
+ vLLM's tokenize request sets no context limit, so a prompt longer than the
+ context is counted instead of refused.
+ """
+
+ def build_tok_params(self, model_config: Any) -> Any:
+ return replace(super().build_tok_params(model_config), max_total_tokens=None)
+
+
+def chat_request_from_render(request: dict[str, Any]) -> NeMoRLChatCompletionRequest:
+ """The chat request a render produced, rebuilt from its JSON without messages.
+
+ vLLM's tool parser sets ``structured_outputs`` during the render (a forced
+ tool's schema), which the request's own validators reject beside a named
+ ``tool_choice``; it is set back after validation, as the render left it.
+ """
+ structured_outputs = request.get("structured_outputs")
+ chat = NeMoRLChatCompletionRequest.model_validate(
+ {
+ **request,
+ "structured_outputs": None,
+ "messages": [{"role": "user", "content": ""}],
+ }
+ )
+ if structured_outputs is not None:
+ chat.structured_outputs = StructuredOutputsParams(
+ **{k: v for k, v in structured_outputs.items() if not k.startswith("_")}
+ )
+ return chat
+
+
+# This MRO is necessary i.e. NeMoRLOpenAIChatRequestMixin > TokenizeRequest
+class NeMoRLTokenizeChatRequest(
+ NeMoRLOpenAIChatRequestMixin, GymRequestFields, TokenizeChatRequest
+):
+ ng_measure_only: Literal[True] = True
+
+
+NeMoRLTokenizeRequest = Union[TokenizeCompletionRequest, NeMoRLTokenizeChatRequest]
+
+
+def reasoning_ended(
+ request: Any, parser: Any, template_token_ids: list[int]
+) -> Optional[bool]:
+ """Whether the model's reasoning is over before this call generates.
+
+ vLLM's rule (its chat serving's), read from the call's template render and
+ never from a spliced prompt: the model's own earlier turns still hold the
+ end-of-reasoning tokens the template drops from history, and they say
+ nothing about this turn. None without a reasoning parser.
+ """
+ if not request.include_reasoning or request._grammar_from_parser:
+ # A Mistral grammar from the parser has its own optional think rule.
+ return True
+ if parser is not None and parser.reasoning_parser is not None:
+ return parser.is_reasoning_end(template_token_ids)
+ return None
+
+
+class NeMoRLOpenAIServingMixin:
+ """The chat preprocessing NeMo RL adds: prefix splice and token capture."""
+
+ async def _render_chat_call(
+ self, request, messages, render_kwargs: dict[str, Any], *, admission
+ ) -> "RenderedChat":
+ """Render a chat call and decide how its capture prefix is served.
+
+ The steps every host of the chat path shares, before any TQ read: the
+ render with the checkpoint's template and processor, the capture
+ decision for a Gym candidate, and, when a prefix will be
+ spliced, the render of the history up to the last assistant turn that
+ the splice cuts against. A chat request renders with an output budget
+ of 1, so vLLM's pre-tokenization length check passes; start_capture sets the
+ budget clamped to the final prompt.
+ """
+ request_max_tokens = None
+ if isinstance(request, ChatCompletionRequest):
+ request_max_tokens = (
+ request.max_completion_tokens
+ if request.max_completion_tokens is not None
+ else request.max_tokens
+ )
+ if request_max_tokens is not None:
+ set_max_tokens(request, 1)
+ for message in messages:
+ if message.get("tool_calls"):
+ message["tool_calls"] = list(message["tool_calls"])
+ messages_for_replace_prefix_tokens = deepcopy(messages)
+
+ # vLLM 0.33 turns parallel_tool_calls=False into a grammar that
+ # masks sampling after the first tool call, so the logprobs it
+ # returns no longer match the trainer's unmasked forward pass.
+ # Hide the flag from the parser only: vLLM still trims the
+ # response to the first tool call, as it did before 0.33.
+ single_tool_call = getattr(request, "parallel_tool_calls", None) is False
+ if single_tool_call:
+ request.parallel_tool_calls = None
+ try:
+ conversation, (engine_prompt,) = await super().preprocess_chat(
+ request=request, messages=messages, **render_kwargs
+ )
+ except (ValueError, VLLMValidationError) as e:
+ if "maximum context length" in str(e):
+ LOGGER.warning("Prompt exceeds max_model_len: %s", e)
+ raise
+ finally:
+ if single_tool_call:
+ request.parallel_tool_calls = False
+
+ ended = None
+ if isinstance(request, ChatCompletionRequest):
+ ended = reasoning_ended(
+ request,
+ self._request_parser(request, render_kwargs),
+ engine_prompt["prompt_token_ids"],
+ )
+
+ # Token capture: the admission is built once by the host and resolved
+ # here, before branching; the ``ng_capture`` dict is never mutated.
+ serving_admission = admission
+ verify_retained_media = False
+ if admission is not None and (
+ admission.mode == "candidate" or admission.request_replay is not None
+ ):
+ # Inline assistant token fields cannot override RL's decision.
+ request.required_prefix_token_ids = None
+ decision = resolve_capture_input(
+ admission,
+ messages=messages_for_replace_prefix_tokens,
+ conversation=conversation,
+ renderer_is_hf=type(self.renderer) is HfRenderer,
+ )
+ admission = decision.storage
+ serving_admission = decision.serving
+ verify_retained_media = decision.verify_retained_media
+
+ history_token_ids = None
+ if getattr(request, "required_prefix_token_ids", None) is not None or (
+ serving_admission is not None and serving_admission.mode == "token_in"
+ ):
+ # The splice cuts the full render at the turn this history ends.
+ last_assistant_message_idx = None
+ for i in reversed(range(len(messages_for_replace_prefix_tokens))):
+ if messages_for_replace_prefix_tokens[i]["role"] == "assistant":
+ last_assistant_message_idx = i
+ break
+ if last_assistant_message_idx is None:
+ messages_to_last_assistant_message = messages_for_replace_prefix_tokens
+ else:
+ messages_to_last_assistant_message = messages_for_replace_prefix_tokens[
+ : last_assistant_message_idx + 1
+ ]
+ modified_request = request.model_copy(
+ update={"add_generation_prompt": False}
+ )
+ _, (history_prompt,) = await super().preprocess_chat(
+ request=modified_request,
+ messages=messages_to_last_assistant_message,
+ **render_kwargs,
+ )
+ history_token_ids = history_prompt["prompt_token_ids"]
+
+ return RenderedChat(
+ template_token_ids=list(engine_prompt["prompt_token_ids"]),
+ history_token_ids=history_token_ids,
+ required_prefix_token_ids=getattr(
+ request, "required_prefix_token_ids", None
+ ),
+ admission=admission,
+ serving_admission=serving_admission,
+ verify_retained_media=verify_retained_media,
+ request_max_tokens=request_max_tokens,
+ reasoning_ended=ended,
+ conversation=conversation,
+ engine_prompt=engine_prompt,
+ )
+
+ def _request_parser(self, request, render_kwargs: dict[str, Any]) -> Any:
+ """The call's parser, built as vLLM's chat serving builds its own.
+
+ None when no reasoning parser is configured: nothing reads it then.
+ """
+ parser_cls = render_kwargs["parser"]
+ if parser_cls is None or parser_cls.reasoning_parser_cls is None:
+ return None
+ chat_params = request.build_chat_params(
+ render_kwargs["default_template"],
+ render_kwargs["default_template_content_format"],
+ ).with_defaults(render_kwargs["default_template_kwargs"])
+ return parser_cls(
+ self.renderer.get_tokenizer(),
+ request.tools,
+ chat_template_kwargs=chat_params.chat_template_kwargs,
+ model_config=self.model_config,
+ )
+
+ # vLLM 0.25 moved chat preprocessing to
+ # OnlineRenderer.preprocess_chat (tool_parser/reasoning_parser were
+ # folded into a single `parser`), so this override now applies via
+ # the renderer subclass.
+ async def preprocess_chat(
+ self,
+ request,
+ messages,
+ default_template,
+ default_template_content_format,
+ default_template_kwargs,
+ tool_dicts=None,
+ parser=None,
+ *,
+ skip_mm_cache: bool = False,
+ ):
+ host = self._host
+ admission = host._capture_admission(request)
+ # #4124: processor-only cache reads retain concrete pixels even
+ # when the engine's sender cache would return references.
+ if host._capture_media and admission is not None:
+ skip_mm_cache = True
+
+ rendered = await self._render_chat_call(
+ request,
+ messages,
+ dict(
+ default_template=default_template,
+ default_template_content_format=default_template_content_format,
+ default_template_kwargs=default_template_kwargs,
+ tool_dicts=tool_dicts,
+ parser=parser,
+ skip_mm_cache=skip_mm_cache,
+ ),
+ admission=admission,
+ )
+ if isinstance(request, NeMoRLChatCompletionRequest):
+ # The request's SamplingParams carry it to the engine client.
+ request._template_reasoning_ended = rendered.reasoning_ended
+ plan = await prepare_prompt(
+ rendered,
+ resolve_prefix=host._resolve_admission_prefix,
+ turn_end_token_ids=turn_end_token_ids(
+ self._model_stop_token_ids, self.renderer.tokenizer
+ ),
+ max_model_len=self.model_config.max_model_len,
+ # /tokenize counts a prompt of any length, so only chat calls clamp.
+ measure_only=not isinstance(request, NeMoRLChatCompletionRequest),
+ tokenizer=self.renderer.tokenizer,
+ )
+ await start_capture(
+ host,
+ request,
+ rendered.engine_prompt,
+ plan,
+ max_model_len=self.model_config.max_model_len,
+ )
+ return rendered.conversation, [rendered.engine_prompt]
+
+
+@dataclass(frozen=True)
+class RenderedChat:
+ """One chat call rendered for the engine, before its prefix is spliced in.
+
+ A :class:`~nemo_rl.models.generation.vllm.prompt_and_capture.RenderedPrompt`.
+ """
+
+ template_token_ids: list[int]
+ # The render up to the last assistant turn, when a prefix will be spliced.
+ history_token_ids: Optional[list[int]]
+ # The request's own prefix (Gym's echo) after the capture decision.
+ required_prefix_token_ids: Optional[list[int]]
+ # What capture stores, after the candidate decision; None without capture.
+ admission: Any
+ # How the prompt is served: "token_in" reads the prefix from TQ.
+ serving_admission: Any
+ verify_retained_media: bool
+ # The output budget the request asked for, before the render lowered it.
+ request_max_tokens: Optional[int]
+ # Whether reasoning is over before the call generates (reasoning_ended).
+ reasoning_ended: Optional[bool]
+ conversation: list[Any]
+ # The full render, multimodal inputs included; its ids are the template's.
+ engine_prompt: dict[str, Any]
+
+
+class NeMoRLOnlineRenderer(NeMoRLOpenAIServingMixin, OnlineRenderer):
+ def __init__(
+ self, *args, host: Any, model_stop_token_ids: frozenset[int], **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self._host = host
+ # vLLM adds the generation config's EOS to every request's stop set
+ # regardless of the configured stops, so a turn can end on either.
+ self._model_stop_token_ids = model_stop_token_ids
+
+
+class FrontendRenderer(NeMoRLOpenAIServingMixin, OnlineRenderer):
+ """The render step for NeMo RL's chat engine in Dynamo's frontend.
+
+ Runs only the shared step (:meth:`_render_chat_call`): the frontend reads
+ the TQ prefix and splices, and the engine captures. Media is always
+ processed here, never served from a processor cache, since the engine runs
+ on the pixels the frontend sends.
+ """
+
+ # The render the overridden preprocess_chat is asked for, and its result.
+ _admission: Any = None
+ _rendered: Any = None
+
+ async def render_for_frontend(self, request, *, admission) -> "RenderedChat":
+ """Render ``request``; the caller restores its lowered output budget."""
+ if self.use_harmony:
+ # vLLM renders gpt-oss's Harmony format without preprocess_chat, so
+ # NeMo RL's render, capture decision and splice would never run.
+ raise ValueError(
+ "gpt-oss (Harmony) chat calls are not served through NeMo RL's "
+ "Dynamo chat engine; use policy.generation.dynamo=false"
+ )
+ self._admission = admission
+ result = await self.render_chat(request, skip_mm_cache=True)
+ if not isinstance(result, tuple):
+ raise ValueError(result.error.message)
+ return self._rendered
+
+ async def preprocess_chat(
+ self,
+ request,
+ messages,
+ default_template,
+ default_template_content_format,
+ default_template_kwargs,
+ tool_dicts=None,
+ parser=None,
+ *,
+ skip_mm_cache: bool = False,
+ ):
+ self._rendered = await self._render_chat_call(
+ request,
+ messages,
+ dict(
+ default_template=default_template,
+ default_template_content_format=default_template_content_format,
+ default_template_kwargs=default_template_kwargs,
+ tool_dicts=tool_dicts,
+ parser=parser,
+ skip_mm_cache=True,
+ ),
+ admission=self._admission,
+ )
+ return self._rendered.conversation, [self._rendered.engine_prompt]
+
+
+_FULL_GENERATOR_SIGNATURE = _full_generator_signature(
+ OpenAIServingChat.chat_completion_full_generator
+)
+
+
+class ResumedCallReplyMixin:
+ """vLLM's reply to a call resumed from a checkpoint cut, as the whole call's.
+
+ The engine ran the restored tokens as prompt and generated only the rest.
+ Before vLLM parses the reply, the restored text goes in front of the final
+ output and the parser sees the restored tokens first; after, usage counts
+ them as completion. It reads the request's restore fields, which
+ ``resolve_restored_call`` sets on the worker and the render pool sets from
+ the engine's reply. The native serving chat and the render pool's both use
+ it.
+ """
+
+ async def chat_completion_full_generator(
+ self,
+ request,
+ result_generator,
+ *args,
+ **kwargs,
+ ):
+ restored_token_ids = request._restored_generation_token_ids
+ engine_results = result_generator
+
+ async def restored_results():
+ final_res = None
+ async for res in engine_results:
+ final_res = res
+ if final_res is None:
+ return
+ # Before vLLM parses the reply, so tool calls and reasoning are
+ # parsed from the whole call's text.
+ restore_reply_text(
+ request,
+ final_res,
+ prefix_token_ids=restored_token_ids,
+ engine_prompt_token_ids=request._restored_prompt_token_ids,
+ tokenizer=self.renderer.tokenizer,
+ )
+ yield final_res
+
+ if request._restored_prompt_token_ids:
+ result_generator = restored_results()
+ # Bound by name: vLLM passes request_metadata by position and
+ # parser by keyword, and either may move.
+ bound = _FULL_GENERATOR_SIGNATURE.bind_partial(
+ self, request, result_generator, *args, **kwargs
+ )
+ if bound.arguments.get("parser") is not None:
+ bound.arguments["parser"] = wrap_parser_for_restore(
+ bound.arguments["parser"], restored_token_ids
+ )
+ response = await super().chat_completion_full_generator(
+ *bound.args[1:], **bound.kwargs
+ )
+ if not isinstance(response, ChatCompletionResponse):
+ return response
+ terminal = request._restored_prefix_terminal
+ if not isinstance(terminal, _RestoredPrefixTerminal):
+ terminal = None
+ if restore_reply_usage(
+ response,
+ restored_token_count=len(restored_token_ids),
+ terminal=terminal,
+ ):
+ request_metadata = bound.arguments.get("request_metadata")
+ if request_metadata is not None:
+ request_metadata.final_usage_info = response.usage
+ return response
+
+
+class NeMoRLOpenAIServingChatMixin:
+ async def chat_completion_full_generator(
+ self,
+ request,
+ result_generator,
+ *args,
+ **kwargs,
+ ):
+ host = self._host
+ return_as_token_id = (
+ request.return_tokens_as_token_ids
+ if request.return_tokens_as_token_ids is not None
+ else self.return_tokens_as_token_ids
+ )
+
+ final_res = None
+ capture_routes = request in host._capture_registry
+ cancelled = False
+
+ def _without_routes(res):
+ # vLLM encodes choice-level routes in its response. Keep
+ # engine arrays for our native capture, and hide them only
+ # in the superclass's shallow view.
+ res = copy.copy(res)
+ res.outputs = [copy.copy(output) for output in res.outputs]
+ for output in res.outputs:
+ output.routed_experts = None
+ return res
+
+ async def capture_result_generator():
+ nonlocal final_res, cancelled
+ try:
+ async for res in result_generator:
+ # Recorded before asking for the next output, which
+ # releases vLLM's finished state for this request.
+ host._prefix_recovery.observe(request, res)
+ final_res = res
+ if final_res is None:
+ return
+ yield (_without_routes(final_res) if capture_routes else final_res)
+ except asyncio.CancelledError:
+ cancelled = True
+ raise
+
+ # Capture stages the raw arrays below and strips OpenAI logprobs
+ # after the terminal PUT. Skip only their object construction;
+ # changing request.logprobs would also change engine sampling.
+ bypass_logprobs = bool(
+ not request.stream
+ and request.logprobs
+ and return_as_token_id
+ and request in host._capture_registry
+ )
+ with self._bypass_chat_logprobs(enabled=bypass_logprobs):
+ response = await super().chat_completion_full_generator(
+ request, capture_result_generator(), *args, **kwargs
+ )
+ if cancelled:
+ # vLLM's chat_completion_full_generator catches the
+ # CancelledError that with_cancellation throws into the
+ # handler task on a client disconnect and returns
+ # ErrorResponse("Client disconnected") instead of
+ # re-raising. Nobody reads that response (the wrapper has
+ # already returned), and letting it through would make the
+ # route label the capture "error_response". Re-raise so the
+ # task finishes cancelled and the route sees the disconnect.
+ raise asyncio.CancelledError()
+ if not isinstance(response, ChatCompletionResponse) or final_res is None:
+ return response
+
+ if request.logprobs and return_as_token_id:
+ response = attach_token_information_to_chat_response_choices(
+ response,
+ final_res,
+ )
+ if request.ng_capture and host._capture_top_logprobs:
+ # Extraction scales with completion length and K.
+ response = await asyncio.to_thread(
+ attach_sampler_topk_logprobs_to_chat_response_choices,
+ response,
+ final_res,
+ top_k=host._capture_top_logprobs,
+ )
+
+ if host._return_routed_experts_enabled():
+ response = attach_routed_experts_to_chat_response_choices(
+ response,
+ final_res,
+ device=torch.device("cpu"),
+ logger=LOGGER,
+ routed_experts_dtype=host.routed_experts_dtype,
+ encode_for_wire=not capture_routes,
+ )
+
+ return response
+
+
+class NeMoRLOpenAIServingChat(
+ NeMoRLOpenAIServingChatMixin,
+ CapturedChatLogprobsMixin,
+ ResumedCallReplyMixin,
+ OpenAIServingChat,
+):
+ def __init__(self, *args, host: Any, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._host = host
+
+
+class FrontendServingChat(ResumedCallReplyMixin, OpenAIServingChat):
+ """The reply step for NeMo RL's chat engine in Dynamo's frontend.
+
+ Builds the reply from the engine's output and never generates.
+ """
+
+
+# Tokenize path delegates to OnlineRenderer.preprocess_chat,
+# where the prefix-token override lives.
+class NeMoRLServingTokenization(ServingTokenization):
+ pass
+
+
+@dataclass(frozen=True)
+class ChatServer:
+ """The serving objects behind /v1/chat/completions and /tokenize."""
+
+ online_renderer: NeMoRLOnlineRenderer
+ serving_chat: NeMoRLOpenAIServingChat
+ serving_tokenization: NeMoRLServingTokenization
+
+
+def register_parser_plugins(cfg: dict[str, Any]) -> None:
+ """Import the tool and reasoning parser plugins ``policy.generation`` names."""
+ maybe_tool_parser_plugin = cfg["vllm_cfg"].get("tool_parser_plugin")
+ if maybe_tool_parser_plugin:
+ ToolParserManager.import_tool_parser(maybe_tool_parser_plugin)
+
+ maybe_reasoning_parser_plugin = cfg["vllm_cfg"].get("reasoning_parser_plugin")
+ if maybe_reasoning_parser_plugin:
+ ReasoningParserManager.import_reasoning_parser(maybe_reasoning_parser_plugin)
+
+
+def serving_chat_settings(cfg: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]:
+ """The chat serving's settings and default chat-template kwargs.
+
+ One copy for every host of the chat path, so each renders and parses
+ alike. Returns the ``OpenAIServingChat`` kwargs (template loaded,
+ ``default_chat_template_kwargs`` normalized) and those template kwargs.
+ """
+ serving_chat_default_kwargs: dict[str, Any] = {
+ "response_role": "assistant",
+ "request_logger": None,
+ "chat_template": None,
+ "chat_template_content_format": "auto",
+ "enable_auto_tools": True,
+ }
+ serving_chat_kwargs = serving_chat_default_kwargs | cfg["vllm_cfg"].get(
+ "http_server_serving_chat_kwargs", dict()
+ )
+ # The embedded server is constructed directly instead of through
+ # vLLM's CLI, where chat-template file paths are normally loaded.
+ # OnlineRenderer expects literal Jinja content; passing a path makes
+ # Transformers render the path itself and drops multimodal
+ # placeholders such as .
+ configured_chat_template = serving_chat_kwargs.get("chat_template")
+ if configured_chat_template is not None:
+ serving_chat_kwargs["chat_template"] = load_chat_template(
+ configured_chat_template
+ )
+ # Recipes may name the parameter either way: ``default_chat_template_kwargs``
+ # is vLLM's own spelling, ``chat_template_kwargs`` is accepted for recipes
+ # written against the older name. Normalize onto the native key rather
+ # than popping it: OnlineRenderer, OpenAIServingChat and ServingTokenization
+ # each keep their *own* copy and read it independently -- the chat serving
+ # builds its reasoning parser from it, and the tokenize path passes its own
+ # into preprocess_chat -- so the renderer's copy does not reach either.
+ # vLLM's api_server hands the same value to all three for that reason.
+ #
+ # Popped separately, not `A or B`: short-circuiting on a truthy A would
+ # leave B in the bag and OpenAIServingChat(**kwargs) would reject it.
+ _legacy_chat_template_kwargs = serving_chat_kwargs.pop("chat_template_kwargs", None)
+ if serving_chat_kwargs.get("default_chat_template_kwargs") is None:
+ serving_chat_kwargs["default_chat_template_kwargs"] = (
+ _legacy_chat_template_kwargs
+ )
+ default_chat_template_kwargs: dict[str, Any] = (
+ serving_chat_kwargs["default_chat_template_kwargs"] or {}
+ )
+ return serving_chat_kwargs, default_chat_template_kwargs
+
+
+def build_chat_server(
+ host: Any, *, engine_client: Any, model_config: Any, cfg: dict[str, Any]
+) -> ChatServer:
+ """Build the chat serving the worker's HTTP server answers with.
+
+ Args:
+ host: The vLLM worker; it owns token capture.
+ engine_client: The engine the chat serving generates with.
+ model_config: The engine's vLLM ``ModelConfig``.
+ cfg: ``policy.generation``.
+ """
+ register_parser_plugins(cfg)
+
+ base_model_paths = [
+ BaseModelPath(
+ name=model_config.served_model_name, model_path=model_config.model
+ ),
+ BaseModelPath(name=model_config.model, model_path=model_config.model),
+ ]
+
+ openai_serving_models_kwargs = dict(
+ engine_client=engine_client,
+ base_model_paths=base_model_paths,
+ lora_modules=None,
+ )
+ openai_serving_models = OpenAIServingModels(**openai_serving_models_kwargs)
+
+ serving_chat_kwargs, default_chat_template_kwargs = serving_chat_settings(cfg)
+ online_renderer = NeMoRLOnlineRenderer(
+ model_config=engine_client.model_config,
+ renderer=engine_client.renderer,
+ request_logger=serving_chat_kwargs["request_logger"],
+ chat_template=serving_chat_kwargs["chat_template"],
+ chat_template_content_format=serving_chat_kwargs[
+ "chat_template_content_format"
+ ],
+ enable_auto_tools=serving_chat_kwargs["enable_auto_tools"],
+ # Keep the renderer's parser consistent with any parser overrides
+ # passed to OpenAIServingChat via http_server_serving_chat_kwargs.
+ tool_parser=serving_chat_kwargs.get("tool_parser"),
+ reasoning_parser=serving_chat_kwargs.get("reasoning_parser"),
+ # vLLM merges these into every render, with request-supplied keys
+ # winning (preprocess_chat's default_template_kwargs). The renderer
+ # is shared by /v1/chat/completions and /tokenize, so setting it
+ # here keeps the two endpoints rendering identical prompts.
+ default_chat_template_kwargs=default_chat_template_kwargs,
+ host=host,
+ model_stop_token_ids=model_stop_token_ids(model_config, cfg),
+ )
+ serving_chat_kwargs.update(
+ dict(
+ engine_client=engine_client,
+ models=openai_serving_models,
+ online_renderer=online_renderer,
+ return_tokens_as_token_ids=True,
+ )
+ )
+ openai_serving_chat = NeMoRLOpenAIServingChat(**serving_chat_kwargs, host=host)
+
+ serving_tokenization_kwargs = dict(
+ request_logger=serving_chat_kwargs["request_logger"],
+ chat_template=serving_chat_kwargs["chat_template"],
+ chat_template_content_format=serving_chat_kwargs[
+ "chat_template_content_format"
+ ],
+ models=serving_chat_kwargs["models"],
+ online_renderer=online_renderer,
+ # ServingTokenization reads its own copy in preprocess_chat rather
+ # than the renderer's, so /tokenize would otherwise render with {}
+ # and diverge from /v1/chat/completions under multi-turn.
+ default_chat_template_kwargs=default_chat_template_kwargs,
+ )
+ openai_serving_tokenization = NeMoRLServingTokenization(
+ **serving_tokenization_kwargs
+ )
+ return ChatServer(online_renderer, openai_serving_chat, openai_serving_tokenization)
+
+
+def _error_response(
+ failure: ErrorReply, headers: Optional[dict[str, str]]
+) -> JSONResponse:
+ """The OpenAI error body of a call refused before its reply."""
+ return JSONResponse(
+ content={
+ "error": {
+ "message": failure.message,
+ "type": failure.error_type,
+ "param": failure.param,
+ "code": failure.code,
+ }
+ },
+ status_code=failure.status,
+ headers=headers or None,
+ )
+
+
+CHAT_COMPLETIONS_PATH = "/v1/chat/completions"
+
+
+def register_routes(
+ app: FastAPI, server: ChatServer, *, host: Any, generation_config: dict[str, Any]
+) -> None:
+ """Add /v1/chat/completions and /tokenize to ``app``."""
+ openai_serving_chat = server.serving_chat
+ openai_serving_tokenization = server.serving_tokenization
+
+ @app.exception_handler(VLLMClientError)
+ async def refuse_unparsed_request(raw_request: Request, error: VLLMClientError):
+ # vLLM raises some request errors from the body's validators, before any
+ # route runs; unhandled, FastAPI answers them with a 500.
+ failure = error_reply(error)
+ if failure is None:
+ raise error
+ headers = None
+ if failure.acknowledge and raw_request.url.path == CHAT_COMPLETIONS_PATH:
+ # No capture began: the refused call is named by the raw body.
+ headers = host._context_refusal_headers(
+ GymRequestFields.model_validate_json(await raw_request.body())
+ )
+ return _error_response(failure, headers)
+
+ class HostedChatCompletionRequest(NeMoRLChatCompletionRequest):
+ def to_sampling_params(self, *args, **kwargs):
+ sampling_params = super().to_sampling_params(*args, **kwargs)
+ if self.ng_capture is not None:
+ host._prefix_recovery.on_sampling_params(self, sampling_params)
+ if self._template_reasoning_ended is not None:
+ # vLLM's serving reads its own from the spliced prompt; the
+ # worker's engine client hands the engine this one instead.
+ sampling_params.extra_args = {
+ **(sampling_params.extra_args or {}),
+ REASONING_ENDED_ARG: self._template_reasoning_ended,
+ }
+ return sampling_params
+
+ register_generation_cut_route(app, lambda: host._prefix_recovery)
+
+ # The create_chat_completion and tokenize methods are taken from vllm/entrypoints/openai/api_server.py
+ @app.post(CHAT_COMPLETIONS_PATH)
+ # Like vLLM's own routes: cancel the handler (and with it the engine
+ # request) when the client disconnects, instead of generating for nobody.
+ @with_cancellation
+ async def create_chat_completion(
+ request: HostedChatCompletionRequest, raw_request: Request
+ ):
+ # Accept train and validation profiles; reject off-policy requests.
+ # vLLM fills an unset top_k/min_p/repetition_penalty from
+ # generation_config.json when vllm_kwargs sets generation_config="auto",
+ # so serve the matched profile's top_k (as
+ # vllm_worker.py::BaseVllmGenerationWorker::_build_sampling_params does)
+ # with min_p and the repetition penalty off; the trainer replays none of
+ # them, nor the penalties or logit bias a request may therefore not set.
+ try:
+ sampling = on_policy_sampling(
+ temperature=request.temperature,
+ top_p=request.top_p,
+ top_k=request.top_k,
+ min_p=request.min_p,
+ repetition_penalty=request.repetition_penalty,
+ presence_penalty=request.presence_penalty,
+ frequency_penalty=request.frequency_penalty,
+ logit_bias=request.logit_bias,
+ generation_config=generation_config,
+ )
+ for field, value in sampling.items():
+ setattr(request, field, value)
+ generator = await openai_serving_chat.create_chat_completion(
+ request, raw_request
+ )
+ except BaseException as error:
+ # A cancel before generation propagates here directly; vLLM swallows a
+ # later one, and NeMoRLOpenAIServingChatMixin's full generator re-raises it.
+ failure = await abort_capture(host, request, error)
+ if failure is None:
+ raise
+ # The acknowledgement goes out after abort_capture drained capture state.
+ return _error_response(
+ failure,
+ host._context_refusal_headers(request) if failure.acknowledge else None,
+ )
+
+ if isinstance(generator, ErrorResponse):
+ await abort_request_capture(host, request, "error_response")
+ return JSONResponse(
+ content=generator.model_dump(), status_code=generator.error.code
+ )
+
+ if isinstance(generator, ChatCompletionResponse):
+ content = model_dump_chat_response_with_dynamic_message_fields(generator)
+ # Token capture: stage the delta and ride the coords on the
+ # response; strips logprobs/ids (no-op when capture is off).
+ content = await save_capture(host, request, content)
+ return JSONResponse(content=content)
+
+ await abort_request_capture(host, request, "streaming_response")
+ return StreamingResponse(content=generator, media_type="text/event-stream")
+
+ @app.post("/tokenize")
+ async def tokenize(request: NeMoRLTokenizeRequest, raw_request: Request):
+ generator = await openai_serving_tokenization.create_tokenize(
+ request, raw_request
+ )
+
+ if isinstance(generator, ErrorResponse):
+ return JSONResponse(
+ content=generator.model_dump(), status_code=generator.error.code
+ )
+ elif isinstance(generator, TokenizeResponse):
+ if (
+ isinstance(request, NeMoRLTokenizeChatRequest)
+ and request.ng_capture is not None
+ ):
+ return JSONResponse(
+ content=GymCallResult(
+ prompt_token_count=generator.count, ng_context_measured=True
+ ).model_dump(exclude_none=True)
+ )
+ return JSONResponse(content=generator.model_dump())
+
+
+class _CleanLoggingFilter(LoggingFilter):
+ def filter(self, record: LogRecord) -> bool:
+ msg = record.getMessage()
+
+ # vLLM does not accept `strict` tool definitions and reporting it to the user is not useful either.
+ return (
+ "Added request" not in msg
+ and "The following fields were present in the request but ignored: {'strict'}"
+ not in msg
+ )
+
+
+# Suppress the noisy vLLM traceback when a prompt exceeds max_model_len.
+# This is expected during multi-turn rollouts; we log a clean one-line
+# warning from preprocess_chat instead.
+class _MaxContextLengthFilter(LoggingFilter):
+ def filter(self, record: LogRecord) -> bool:
+ if record.exc_info and record.exc_info[1]:
+ if "maximum context length" in str(record.exc_info[1]):
+ return False
+ return True
+
+
+def install_log_filters() -> None:
+ """Keep vLLM's per-request and context-overflow noise out of the logs."""
+ print(
+ "Adding a vLLM logging filter so that the logs aren't spammed with not useful messages like `Added request ...`. This is to help errors pop up better and filter out noise."
+ )
+ vllm_async_llm_logger.addFilter(_CleanLoggingFilter())
+ logging.getLogger("vllm.entrypoints.openai.engine.protocol").addFilter(
+ _CleanLoggingFilter()
+ )
+ logging.getLogger("vllm.entrypoints.openai.chat_completion.serving").addFilter(
+ _MaxContextLengthFilter()
+ )
diff --git a/nemo_rl/models/generation/vllm/generation_cut_route.py b/nemo_rl/models/generation/vllm/generation_cut_route.py
index f1b0f43da40..d2d936713c2 100644
--- a/nemo_rl/models/generation/vllm/generation_cut_route.py
+++ b/nemo_rl/models/generation/vllm/generation_cut_route.py
@@ -18,25 +18,38 @@
import secrets
from collections.abc import Callable
+from typing import Any, Protocol
from fastapi import FastAPI, Header, HTTPException
-from nemo_rl.models.generation.prefix_recovery import NoPrefixRecovery, PrefixRecovery
-
# Gym owns this route (nemo_gym._checkpoint.generation_cut.GENERATION_CUT_ROUTE),
# but nemo_gym is an optional extra. A drifted copy would 404 every cut, which Gym
# treats as "regenerate", so test_generation_cut_route_matches_gym pins it.
GENERATION_CUT_ROUTE = "/ng-control/v1/generation-cut"
+class GenerationCutBackend(Protocol):
+ """What runs a cut: a worker's prefix recovery, or the Dynamo frontend's fan-out."""
+
+ @property
+ def control_token(self) -> str | None:
+ """The bearer token a cut must carry; None refuses every cut."""
+ ...
+
+ async def run_cut(self, inventory: Any) -> Any:
+ """Cut the calls Gym's inventory names and return Gym's receipt."""
+ ...
+
+
def register_generation_cut_route(
- app: FastAPI, recovery: Callable[[], PrefixRecovery | NoPrefixRecovery]
+ app: FastAPI, recovery: Callable[[], GenerationCutBackend]
) -> None:
- """Serve Gym's generation-cut call from the worker's prefix recovery.
+ """Serve Gym's generation-cut call from the host's cut backend.
``recovery`` returns the worker's current ``PrefixRecovery`` or
- ``NoPrefixRecovery``. It is a callable because the server starts before
- ``setup_token_capture`` builds the recovery object.
+ ``NoPrefixRecovery``, or the Dynamo frontend's fan-out to its engines. It
+ is a callable because the server starts before ``setup_token_capture``
+ builds the recovery object.
"""
@app.post(GENERATION_CUT_ROUTE)
diff --git a/nemo_rl/models/generation/vllm/prompt_and_capture.py b/nemo_rl/models/generation/vllm/prompt_and_capture.py
new file mode 100644
index 00000000000..ee41029e74e
--- /dev/null
+++ b/nemo_rl/models/generation/vllm/prompt_and_capture.py
@@ -0,0 +1,646 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""The steps of one Gym chat call after its render, shared by every serving path.
+
+1. :func:`prepare_prompt` turns a render into a :class:`PreparedPrompt`: the
+ token-capture prefix read from TQ, the splice, and the output budget.
+2. On the engine's worker, :func:`start_capture` takes the prepared prompt to
+ ``begin_call``: a restore from a checkpoint cut, the budget, the media
+ snapshot, the placeholder remap and the final prompt; after generation,
+ :func:`save_capture` stages the call in one TQ write.
+3. :func:`abort_capture` fails the capture of a call that raised, and
+ :func:`error_reply` says how to answer it.
+
+The native vLLM server runs all of them in the vLLM worker. Behind Dynamo,
+the frontend's chat engine prepares the prompt (its KV router needs the final
+token ids before it picks an engine) and the engine's adapter runs the rest,
+with the prepared prompt sent across as JSON (:meth:`PreparedPrompt.to_json`).
+A restore is resolved on the worker: its weight-version check needs the
+engine's own.
+
+This module imports vLLM; import it only where vLLM is installed.
+"""
+
+import asyncio
+import logging
+from dataclasses import dataclass
+from typing import Any, Callable, Collection, Optional, Protocol
+
+from vllm.entrypoints.serve.exception_handling.error_response import (
+ create_error_response,
+)
+from vllm.exceptions import GracefulHTTPError, VLLMClientError, VLLMValidationError
+
+from nemo_rl.data.captured_media import MediaCaptureRejected
+from nemo_rl.models.generation.openai_server_utils import (
+ PrefixSplice,
+ splice_prefix_tokens,
+)
+from nemo_rl.models.generation.prefix_recovery.restore import (
+ _classify_restored_prefix_terminal,
+ _RestoredPrefixTerminal,
+ mark_restored_terminal,
+ plan_restore,
+)
+from nemo_rl.models.generation.vllm.utils import (
+ OffPolicySamplingError,
+ remap_multimodal_placeholders,
+)
+
+LOGGER = logging.getLogger(__name__)
+
+
+def set_max_tokens(request: Any, max_tokens: int) -> None:
+ """Set the request's max output tokens.
+
+ Mutates the request in place. Handles both max_completion_tokens (newer OpenAI API)
+ and max_tokens (deprecated but still supported by vLLM). A request
+ that set neither gets max_tokens: otherwise vLLM would fall back to
+ the generation config's budget, e.g. for a restored call whose
+ remaining budget is smaller.
+ """
+ if request.max_completion_tokens is not None:
+ request.max_completion_tokens = max_tokens
+ else:
+ request.max_tokens = max_tokens
+
+
+def clamp_max_tokens(
+ request_max_tokens: Optional[int], prompt_len: int, max_model_len: int
+) -> Optional[int]:
+ """The output budget that keeps input + output within ``max_model_len``.
+
+ Refuses a prompt that leaves no room for output, whether or not the call set
+ a budget, and lowers the budget only when it did.
+ """
+ remaining = max_model_len - prompt_len
+ if remaining <= 0:
+ # preserve the literal "context length" in this message to match Gym's overflow handling
+ message = (
+ f"Prompt length ({prompt_len}) fills or exceeds "
+ f"this model's maximum context length ({max_model_len}). "
+ f"No room for output tokens."
+ )
+ LOGGER.warning("Prompt exceeds max_model_len: %s", message)
+ raise VLLMValidationError(message, parameter="input_tokens", value=prompt_len)
+ if request_max_tokens is None:
+ return None
+ return min(request_max_tokens, remaining)
+
+
+@dataclass(frozen=True)
+class RestoredCall:
+ """A call that resumes from the tokens a checkpoint cut saved."""
+
+ # The verified saved rows, as capture's begin_call takes them.
+ saved: Any
+ # The engine prompt: the capture prefix, then the saved rows.
+ prompt_token_ids: list[int]
+ # The saved rows' generated tokens alone.
+ resumed_generation_token_ids: list[int]
+ # The whole output budget, as the cut call resolved it.
+ output_limit: int
+ # What is left of that budget; None keeps the request's own.
+ max_tokens: Optional[int]
+
+
+async def resolve_restored_call(
+ host: Any, request: Any, serving_admission: Any, prefix_token_ids: list[int]
+) -> tuple[Any, Optional[RestoredCall]]:
+ """Resolve the prefix a checkpoint cut saved for this call.
+
+ Returns the admission to serve and the restore. A prefix the worker cannot
+ use comes back as the admission without its cut and None, so the call
+ regenerates. For a restore, the request gets its restore fields (which the
+ reply reads) and its remaining ``min_tokens``.
+ """
+ if serving_admission.generation_cut is None:
+ return serving_admission, None
+ # Off-loop: the saved rows are a blocking TQ read.
+ resolved_admission, saved = await asyncio.to_thread(
+ host._prefix_recovery.resolve_restore, serving_admission, prefix_token_ids
+ )
+ if saved is None:
+ return resolved_admission, None
+ original_min_tokens = getattr(request, "min_tokens", None)
+ plan = plan_restore(
+ prefix_token_ids=prefix_token_ids,
+ saved=saved,
+ continuation=resolved_admission.generation_cut,
+ min_tokens=original_min_tokens,
+ )
+ request._restored_generation_token_ids = plan.resumed_generation_token_ids
+ request._restored_prompt_token_ids = tuple(plan.engine_prefix_token_ids)
+ request._original_min_tokens = original_min_tokens
+ if plan.min_tokens is not None:
+ request.min_tokens = plan.min_tokens
+ return resolved_admission, RestoredCall(
+ saved=saved,
+ prompt_token_ids=plan.engine_prefix_token_ids,
+ resumed_generation_token_ids=list(plan.resumed_generation_token_ids),
+ output_limit=plan.output_limit,
+ max_tokens=plan.max_tokens,
+ )
+
+
+def classify_restored_terminal(
+ restored: RestoredCall,
+ prompt_token_ids: list[int],
+ admission: Any,
+ *,
+ max_model_len: int,
+) -> Optional[_RestoredPrefixTerminal]:
+ """The terminal of a saved prefix that already finished its call, else None."""
+ continuation = admission.generation_cut
+ try:
+ return _classify_restored_prefix_terminal(
+ prompt_token_ids=prompt_token_ids,
+ prefix_token_count=continuation.prefix_token_count,
+ requested_output_tokens=restored.output_limit,
+ model_max_tokens=max_model_len,
+ terminal_finish_reason=continuation.terminal_finish_reason,
+ terminal_stop_reason=continuation.terminal_stop_reason,
+ )
+ except ValueError as error:
+ raise VLLMValidationError(
+ str(error),
+ parameter="generation_prefix",
+ value=continuation.prefix_token_count,
+ ) from error
+
+
+def serve_restored_terminal(
+ request: Any,
+ engine_prompt: Any,
+ terminal: _RestoredPrefixTerminal,
+ admission: Any,
+ *,
+ max_model_len: int,
+) -> None:
+ """Have the engine client answer a finished prefix without decoding.
+
+ Call it in the task that submits the call to the engine: the marker is a
+ context variable.
+ """
+ # One token is the smallest budget vLLM accepts, so a missed marker costs
+ # one token, not the whole call.
+ request.min_tokens = 0
+ set_max_tokens(request, 1)
+ mark_restored_terminal(terminal)
+ prompt_token_ids = engine_prompt["prompt_token_ids"]
+ if len(prompt_token_ids) == max_model_len:
+ engine_prompt["prompt_token_ids"] = prompt_token_ids[:-1]
+ request._restored_prefix_terminal = terminal
+ LOGGER.info(
+ "generation prefix already terminal: "
+ "rollout_id=%s model_call_id=%s prefix_tokens=%d reason=%s",
+ admission.capture_key,
+ admission.model_call_id,
+ terminal.prefix_token_count,
+ terminal.reason,
+ )
+
+
+async def abort_request_capture(host: Any, request: Any, reason: str) -> None:
+ """Fail a request's capture without blocking the event loop.
+
+ An uncontended abort finishes here, so it never queues behind
+ executor work. Only one that must wait out a generation cut or a
+ terminal write holding the call's lifecycle lock moves off the loop.
+ """
+ if host._abort_request_capture(request, reason=reason, blocking=False):
+ return
+ # Shielded: these paths often run after the handler was cancelled,
+ # and a second cancel must not drop the abort before it starts,
+ # leaving the call neither completed nor failed.
+ await asyncio.shield(
+ asyncio.to_thread(host._abort_request_capture, request, reason=reason)
+ )
+
+
+class RenderedPrompt(Protocol):
+ """What :func:`prepare_prompt` reads from a call's render.
+
+ Read-only members, so each path's frozen render dataclass fits.
+ """
+
+ # The full render's token ids, in the template's coordinates.
+ @property
+ def template_token_ids(self) -> list[int]: ...
+
+ # The render up to the last assistant turn, when a prefix will be spliced.
+ @property
+ def history_token_ids(self) -> Optional[list[int]]: ...
+
+ # The request's own prefix (Gym's echo) after the capture decision.
+ @property
+ def required_prefix_token_ids(self) -> Optional[list[int]]: ...
+
+ # Capture admissions: what is stored, and how the prompt is served; None
+ # without capture.
+ @property
+ def admission(self) -> Any: ...
+
+ @property
+ def serving_admission(self) -> Any: ...
+
+ @property
+ def verify_retained_media(self) -> bool: ...
+
+ # The output budget the request asked for; None when it set none.
+ @property
+ def request_max_tokens(self) -> Optional[int]: ...
+
+
+@dataclass(frozen=True)
+class PreparedPrompt:
+ """A call's prompt as the engine runs it, its output budget, and how capture admits it."""
+
+ prompt_token_ids: list[int]
+ # How the model's own prefix replaced the template's history; None if not.
+ splice: Optional[PrefixSplice]
+ # The model's own prefix is the prompt's first ``prefix_len`` ids.
+ prefix_len: Optional[int]
+ # Whether Gym only measures the prompt; such a call never generates.
+ measure_only: bool
+ # The output budget the request asked for, and the one after the clamp;
+ # None when the request set none.
+ request_max_tokens: Optional[int]
+ max_tokens: Optional[int]
+ admission: Any
+ serving_admission: Any
+ verify_retained_media: bool
+
+ @property
+ def begin_call_prefix_token_ids(self) -> Optional[list[int]]:
+ """The prefix ``begin_call`` checks a token-in admission against."""
+ if self.admission is None or self.admission.mode != "token_in":
+ return None
+ return self.prompt_token_ids[: self.prefix_len]
+
+ def to_json(self) -> dict[str, Any]:
+ """As JSON, without the prompt (it rides as the call's token ids)."""
+ return {
+ "splice": None
+ if self.splice is None
+ else [self.splice.model_cut_end, self.splice.template_cut_start],
+ "prefix_len": self.prefix_len,
+ "request_max_tokens": self.request_max_tokens,
+ "max_tokens": self.max_tokens,
+ "admission": _admission_json(self.admission),
+ "serving_admission": _admission_json(self.serving_admission),
+ "verify_retained_media": self.verify_retained_media,
+ }
+
+ @classmethod
+ def from_json(
+ cls,
+ wire: dict[str, Any],
+ *,
+ prompt_token_ids: list[int],
+ parse_admission: Callable[[Any], Any],
+ ) -> "PreparedPrompt":
+ """What :meth:`to_json` sent, for the prompt it rides beside."""
+ splice = None
+ if wire["splice"] is not None:
+ splice = PrefixSplice(prompt_token_ids, *wire["splice"])
+ admission = parse_admission(wire["admission"])
+ # One object when storage and serving are one admission, as natively.
+ serving_admission = admission
+ if wire["serving_admission"] != wire["admission"]:
+ serving_admission = parse_admission(wire["serving_admission"])
+ return cls(
+ prompt_token_ids=prompt_token_ids,
+ splice=splice,
+ prefix_len=wire["prefix_len"],
+ # A measure call is answered where it is prepared.
+ measure_only=False,
+ request_max_tokens=wire["request_max_tokens"],
+ max_tokens=wire["max_tokens"],
+ admission=admission,
+ serving_admission=serving_admission,
+ verify_retained_media=wire["verify_retained_media"],
+ )
+
+
+def _admission_json(admission: Any) -> Optional[dict[str, Any]]:
+ return None if admission is None else admission.model_dump(mode="json")
+
+
+async def prepare_prompt(
+ rendered: RenderedPrompt,
+ *,
+ resolve_prefix: Callable[[Any], list[int]],
+ turn_end_token_ids: Collection[int],
+ max_model_len: int,
+ measure_only: bool,
+ tokenizer: Any = None,
+) -> PreparedPrompt:
+ """Prepare a rendered call's prompt: the ids the engine runs, and its output budget.
+
+ Args:
+ rendered: The call's render.
+ resolve_prefix: Reads a token-in admission's prefix from TQ. It blocks,
+ so it runs off the event loop.
+ turn_end_token_ids: Every id a turn can end on; the splice cuts by them.
+ max_model_len: The engine's context length.
+ measure_only: Whether Gym only measures the prompt. Such a call counts
+ a prompt of any length; a generating call's budget is clamped to
+ the context.
+ tokenizer: Only for the splice's error message; may be None.
+
+ Raises:
+ VLLMValidationError: A generating call's prompt leaves no room for
+ output. Raised before any media check, so the length error is the
+ one the call answers with.
+ """
+ prefix = rendered.required_prefix_token_ids
+ serving = rendered.serving_admission
+ if serving is not None and serving.mode == "token_in":
+ prefix = await asyncio.to_thread(resolve_prefix, serving)
+ splice = None
+ prompt_token_ids = rendered.template_token_ids
+ if prefix is not None:
+ # The render includes the history whenever a prefix will be spliced.
+ assert rendered.history_token_ids is not None
+ splice = splice_prefix_tokens(
+ tokenizer=tokenizer,
+ model_prefix_token_ids=list(prefix),
+ template_prefix_token_ids=rendered.history_token_ids,
+ template_token_ids=rendered.template_token_ids,
+ eos_token_id=turn_end_token_ids,
+ )
+ prompt_token_ids = splice.token_ids
+ max_tokens = None
+ if not measure_only:
+ max_tokens = clamp_max_tokens(
+ rendered.request_max_tokens, len(prompt_token_ids), max_model_len
+ )
+ return PreparedPrompt(
+ prompt_token_ids=prompt_token_ids,
+ splice=splice,
+ prefix_len=None if prefix is None else len(prefix),
+ measure_only=measure_only,
+ request_max_tokens=rendered.request_max_tokens,
+ max_tokens=max_tokens,
+ admission=rendered.admission,
+ serving_admission=rendered.serving_admission,
+ verify_retained_media=rendered.verify_retained_media,
+ )
+
+
+async def start_capture(
+ host: Any,
+ request: Any,
+ engine_prompt: dict[str, Any],
+ plan: PreparedPrompt,
+ *,
+ max_model_len: int,
+) -> None:
+ """Take a prepared prompt to the start of its capture, on the engine's worker.
+
+ In order: resolve a restore from a checkpoint cut (its saved rows replace
+ the prepared prompt), set the output budget, snapshot the call's media for
+ capture, make ``engine_prompt`` the final prompt (its media placeholders
+ moved with the splice), begin the call's capture, and have the engine
+ answer a restored call that had already finished.
+
+ Args:
+ host: The vLLM worker; it owns token capture.
+ request: The chat request the engine serves.
+ engine_prompt: The render's engine input, in the template's
+ coordinates; updated in place.
+ plan: The call's prepared prompt.
+ max_model_len: The engine's context length.
+ """
+ admission = plan.admission
+ serving_admission = plan.serving_admission
+ if admission is not None and admission.mode == "candidate":
+ raise RuntimeError(
+ "a candidate admission reached the engine undecided; NeMo RL's "
+ "render must decide it before capture"
+ )
+ prompt_token_ids = plan.prompt_token_ids
+ splice = plan.splice
+ max_tokens = plan.max_tokens
+ restored = None
+ terminal = None
+ if serving_admission is not None:
+ # A cut this worker cannot use comes back as None: the prepared prompt
+ # stands, and the call regenerates.
+ resolved_admission, restored = await resolve_restored_call(
+ host,
+ request,
+ serving_admission,
+ # The model's own prefix is the prepared prompt's first ids.
+ prompt_token_ids[: plan.prefix_len]
+ if serving_admission.mode == "token_in"
+ else [],
+ )
+ if serving_admission is admission:
+ admission = resolved_admission
+ serving_admission = resolved_admission
+ if restored is not None:
+ # The saved rows are the exact engine prompt at the cut: never spliced or
+ # truncated again (test_vllm_progress_read_layout.py).
+ splice = None
+ prompt_token_ids = restored.prompt_token_ids
+ terminal = classify_restored_terminal(
+ restored, prompt_token_ids, admission, max_model_len=max_model_len
+ )
+ # A restored terminal prefix generates nothing more.
+ max_tokens = None
+ if not plan.measure_only and terminal is None:
+ max_tokens = clamp_max_tokens(
+ restored.max_tokens
+ if restored.max_tokens is not None
+ else plan.request_max_tokens,
+ len(prompt_token_ids),
+ max_model_len,
+ )
+ if max_tokens is not None:
+ set_max_tokens(request, max_tokens)
+ media = None
+ media_admission = admission if splice is None else serving_admission
+ if restored is None and media_admission is not None:
+ # Off-loop: a retained-media check reads staged calls.
+ media = await asyncio.to_thread(
+ host._capture_request_media,
+ engine_prompt,
+ admission=media_admission,
+ splice=splice,
+ verify_retained_media=plan.verify_retained_media,
+ )
+ elif restored is not None:
+ # The saved rows are the prompt, but its media still sits where the
+ # prepared prompt's splice puts it. Off-loop: it hashes the pixels.
+ media = await asyncio.to_thread(
+ host._capture_restored_media,
+ engine_prompt,
+ admission=admission,
+ splice=plan.splice,
+ snapshot=restored.saved,
+ )
+ if splice is not None or restored is not None:
+ # Successful media capture already remapped these ranges. The remap
+ # reads the template's ids, so it runs before they are replaced.
+ if media is None and (placeholders := engine_prompt.get("mm_placeholders")):
+ engine_prompt["mm_placeholders"] = remap_multimodal_placeholders(
+ template_token_ids=engine_prompt["prompt_token_ids"],
+ final_token_ids=prompt_token_ids,
+ mm_placeholders=placeholders,
+ splice=splice,
+ )
+ engine_prompt["prompt_token_ids"] = prompt_token_ids
+ host._begin_request_capture(
+ request,
+ prompt_token_ids,
+ admission=admission,
+ prefix_token_ids=plan.begin_call_prefix_token_ids,
+ media=media,
+ generation_cut=None if restored is None else restored.saved,
+ resumed_generation_token_ids=(
+ None if restored is None else restored.resumed_generation_token_ids
+ ),
+ )
+ if terminal is not None:
+ # The caller's task also submits the call: the engine client answers it.
+ serve_restored_terminal(
+ request, engine_prompt, terminal, admission, max_model_len=max_model_len
+ )
+
+
+async def save_capture(host: Any, request: Any, content: dict) -> dict:
+ """Stage a finished call and ride its coords on ``content``.
+
+ A no-op without capture. Off the loop: the sink write is a blocking TQ
+ round trip.
+ """
+ try:
+ return await asyncio.to_thread(host._finish_request_capture, request, content)
+ except asyncio.CancelledError:
+ # Queued executor work may never start after cancellation.
+ # If staging already claimed the capture, abort is a no-op.
+ await abort_request_capture(host, request, "client_disconnect")
+ raise
+
+
+@dataclass(frozen=True)
+class ErrorReply:
+ """How a call that failed before its reply is answered, on every path."""
+
+ status: int
+ # The OpenAI error body.
+ message: str
+ error_type: str
+ param: Optional[str]
+ code: Any
+ # What the call's capture is failed with.
+ abort_reason: str
+ # Whether Gym is told its capture call was refused before it was staged.
+ acknowledge: bool
+
+
+def error_reply(error: Exception) -> Optional[ErrorReply]:
+ """The answer to a call that raised ``error``; None for a server error (500)."""
+ message = str(error)
+ # Gym detects a context overflow by this text.
+ overflow = "maximum context length" in message
+ if isinstance(error, MediaCaptureRejected):
+ # First, since it is a ValueError too. A stable code lets Gym and the
+ # worker log tell a retained-media re-tile from an engine error.
+ return ErrorReply(
+ status=400,
+ message=message,
+ error_type="invalid_request_error",
+ param="messages",
+ code=error.code,
+ abort_reason=error.code,
+ acknowledge=False,
+ )
+ if isinstance(error, OffPolicySamplingError):
+ return ErrorReply(
+ status=400,
+ message=message,
+ error_type="invalid_request_error",
+ param="sampling",
+ code=400,
+ abort_reason="invalid_request",
+ acknowledge=False,
+ )
+ if isinstance(error, VLLMValidationError):
+ return ErrorReply(
+ status=400,
+ message=message,
+ error_type="invalid_request_error",
+ param=error.parameter,
+ code=400,
+ abort_reason="context_length" if overflow else "invalid_request",
+ acknowledge=True,
+ )
+ if isinstance(error, (VLLMClientError, GracefulHTTPError)):
+ # vLLM's own status: 503 for a full queue, so Gym retries elsewhere.
+ info = create_error_response(error).error
+ return ErrorReply(
+ status=info.code,
+ message=info.message,
+ error_type=info.type,
+ param=info.param,
+ code=info.code,
+ abort_reason=(
+ "invalid_request"
+ if isinstance(error, VLLMClientError)
+ else "engine_error"
+ ),
+ acknowledge=False,
+ )
+ if isinstance(error, ValueError) and overflow:
+ # vLLM's get_max_tokens refuses an over-long prompt with a plain ValueError
+ # ("Input length (N) exceeds ..."); only that refusal settles capture.
+ return ErrorReply(
+ status=400,
+ message=message,
+ error_type="invalid_request_error",
+ param="input_tokens",
+ code=400,
+ abort_reason="context_length",
+ acknowledge=message.startswith("Input length (")
+ and "exceeds model's maximum context length" in message,
+ )
+ return None
+
+
+async def abort_capture(
+ host: Any, request: Any, error: BaseException
+) -> Optional[ErrorReply]:
+ """Fail the capture of a call that raised ``error``; say how to answer it.
+
+ Returns None when the call fails as a server error or a disconnect: the
+ caller re-raises. A no-op on capture once the call was staged.
+ """
+ if isinstance(error, asyncio.CancelledError):
+ await abort_request_capture(host, request, "client_disconnect")
+ return None
+ failure = error_reply(error) if isinstance(error, Exception) else None
+ await abort_request_capture(
+ host, request, "engine_error" if failure is None else failure.abort_reason
+ )
+ if failure is not None and failure.status < 500:
+ LOGGER.warning(
+ "Refused a chat call before it ran (%s): %s",
+ failure.abort_reason,
+ failure.message,
+ )
+ return failure
diff --git a/nemo_rl/models/generation/vllm/utils.py b/nemo_rl/models/generation/vllm/utils.py
index 733c8a31fb6..724f0c3c80c 100644
--- a/nemo_rl/models/generation/vllm/utils.py
+++ b/nemo_rl/models/generation/vllm/utils.py
@@ -12,11 +12,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from collections import defaultdict
+from collections import ChainMap, defaultdict
from collections.abc import Mapping, Sequence
from dataclasses import is_dataclass, replace
from typing import Any, Optional, cast
+import numpy as np
import torch
from nemo_rl.data.multimodal_utils import (
@@ -171,6 +172,18 @@ def _select_flat_logprobs(
)
+def model_stop_token_ids(model_config: Any, cfg: dict[str, Any]) -> frozenset[int]:
+ """The configured stops plus the generation config's EOS, read once per server.
+
+ ``model_config`` is vLLM's ``ModelConfig``; ``cfg`` is ``policy.generation``.
+ """
+ stop_token_ids = set(cfg.get("stop_token_ids") or ())
+ generation_eos = model_config.try_get_generation_config().get("eos_token_id")
+ if isinstance(generation_eos, int):
+ generation_eos = [generation_eos]
+ return frozenset(stop_token_ids | set(generation_eos or ()))
+
+
def assert_reload_refit_config_supported(config: VllmConfig) -> None:
"""Reject pure-config combinations unsupported by vLLM reload refit."""
vllm_cfg = config.get("vllm_cfg")
@@ -489,6 +502,122 @@ def check_sampling_param(
return profile_top_k
+class OffPolicySamplingError(ValueError):
+ """A NeMo-Gym call that would sample off-policy; the trainer replays none of it."""
+
+
+# Sampling a NeMo-Gym call may not change, and the value it is served with: vLLM
+# would otherwise fill min_p and the repetition penalty from the model's
+# generation_config.json, and the trainer replays none of these.
+_FIXED_SAMPLING = {
+ "min_p": 0.0,
+ "repetition_penalty": 1.0,
+ "presence_penalty": 0.0,
+ "frequency_penalty": 0.0,
+}
+
+
+def _f32(value: Optional[float]) -> Optional[float]:
+ return None if value is None else float(np.float32(value))
+
+
+# SamplingParams.extra_args key: a native chat call's reasoning_ended, as
+# NeMo RL read it from the call's template render.
+REASONING_ENDED_ARG = "nemo_rl_reasoning_ended"
+
+
+def take_reasoning_ended(sampling_params: Any, engine_kwargs: dict[str, Any]) -> None:
+ """Hand the engine the ``reasoning_ended`` a call's SamplingParams carry.
+
+ vLLM's chat serving reads its own value from the spliced prompt, whose
+ earlier turns still hold their own end-of-reasoning tokens; this replaces
+ it and takes the argument back out of ``extra_args``. A call that carries
+ none is left as it is.
+ """
+ extra_args = sampling_params.extra_args
+ if not extra_args or REASONING_ENDED_ARG not in extra_args:
+ return
+ engine_kwargs["reasoning_ended"] = extra_args.pop(REASONING_ENDED_ARG)
+ if not extra_args:
+ sampling_params.extra_args = None
+
+
+def on_policy_sampling(
+ *,
+ temperature: Optional[float],
+ top_p: Optional[float],
+ top_k: Optional[int],
+ min_p: Optional[float],
+ repetition_penalty: Optional[float],
+ presence_penalty: Optional[float],
+ frequency_penalty: Optional[float],
+ logit_bias: Optional[Mapping[Any, float]],
+ generation_config: Any,
+) -> dict[str, Any]:
+ """The sampling a NeMo-Gym chat call is served with, on every serving path.
+
+ The request must match the train or validation profile
+ (:func:`check_sampling_param`) and set no sampling the trainer does not
+ replay. Floats are compared at f32: Dynamo's frontend carries them as f32,
+ so 0.7 arrives as 0.69999998.
+
+ Args:
+ temperature, top_p, top_k, min_p, repetition_penalty, presence_penalty,
+ frequency_penalty, logit_bias:
+ The request's values; None (or missing) when it did not set them.
+ generation_config: ``policy.generation``.
+
+ Returns:
+ The ``SamplingParams`` fields to serve with: the matched profile's exact
+ temperature, top_p and top_k, and the fixed min_p and penalties.
+
+ Raises:
+ OffPolicySamplingError: The request would sample off-policy.
+ """
+ # The profile's floats at f32; top_k and val_top_k as configured.
+ profile = ChainMap(
+ {
+ key: _f32(generation_config[key])
+ for key in ("temperature", "top_p", "val_temperature", "val_top_p")
+ },
+ generation_config,
+ )
+ try:
+ served_top_k = check_sampling_param(
+ temperature=_f32(temperature),
+ top_p=_f32(top_p),
+ top_k=top_k,
+ generation_config=profile,
+ )
+ except ValueError as error:
+ raise OffPolicySamplingError(str(error)) from error
+ requested = {
+ "min_p": min_p,
+ "repetition_penalty": repetition_penalty,
+ "presence_penalty": presence_penalty,
+ "frequency_penalty": frequency_penalty,
+ }
+ for key, served in _FIXED_SAMPLING.items():
+ if requested[key] not in (None, served):
+ raise OffPolicySamplingError(
+ f"request {key}={requested[key]} is not supported; leave it unset"
+ )
+ if logit_bias:
+ raise OffPolicySamplingError("request logit_bias is not supported")
+ is_train = (_f32(temperature), _f32(top_p)) == (
+ profile["temperature"],
+ profile["top_p"],
+ )
+ prefix = "" if is_train else "val_"
+ return {
+ "temperature": generation_config[f"{prefix}temperature"],
+ "top_p": generation_config[f"{prefix}top_p"],
+ "top_k": served_top_k,
+ "min_p": _FIXED_SAMPLING["min_p"],
+ "repetition_penalty": _FIXED_SAMPLING["repetition_penalty"],
+ }
+
+
def validate_rollout_prompt(expected: list[int], actual: list[int] | None) -> None:
"""Reject rollouts generated from a different prompt than the learner's."""
if actual != expected:
diff --git a/nemo_rl/models/generation/vllm/video_utils.py b/nemo_rl/models/generation/vllm/video_utils.py
index 4068042d29e..0380912fab3 100644
--- a/nemo_rl/models/generation/vllm/video_utils.py
+++ b/nemo_rl/models/generation/vllm/video_utils.py
@@ -29,6 +29,7 @@
CACHED_VIDEO_FRAME_MANIFEST_MAGIC,
CACHED_VIDEO_FRAME_MANIFEST_MIME,
)
+from nemo_rl.models.generation.vllm.config import VllmConfig, resolve_vllm_video_config
VideoSamplingStyle = Literal["nemotron_vl"]
@@ -518,6 +519,21 @@ def _load_video_frames_torchcodec_with_metadata(
return frames, metadata
+def register_vllm_video_loader(generation_config: VllmConfig) -> None:
+ """Register NeMo RL's video loader when ``vllm_cfg.video`` is set.
+
+ Called by every process that decodes the videos of chat calls: the vLLM
+ engine, and with Dynamo the frontend's render workers, whose pixels the
+ engine then runs.
+ """
+ video_config = resolve_vllm_video_config(generation_config)
+ if video_config is not None:
+ register_torchcodec_vllm_video_loader(
+ sampling_style=video_config.sampling_style,
+ temporal_patch_size=video_config.temporal_patch_size,
+ )
+
+
def register_torchcodec_vllm_video_loader(
*,
sampling_style: VideoSamplingStyle,
diff --git a/nemo_rl/models/generation/vllm/vllm_generation.py b/nemo_rl/models/generation/vllm/vllm_generation.py
index 929605a2444..063893a444b 100644
--- a/nemo_rl/models/generation/vllm/vllm_generation.py
+++ b/nemo_rl/models/generation/vllm/vllm_generation.py
@@ -32,6 +32,8 @@
import numpy as np
import ray
from ray.util.placement_group import PlacementGroup
+from ray.util.queue import Empty
+from ray.util.queue import Queue as RayQueue
from nemo_rl.data.multimodal_utils import VLLM_PROMPT_KEYS
from nemo_rl.distributed.batched_data_dict import BatchedDataDict, SlicedDataDict
@@ -216,22 +218,36 @@ class VllmGeneration(GenerationInterface):
# defaults, so shutdown() also works when __init__ failed before starting them.
_dynamo_runtime: Optional["ManagedDynamoRuntime"] = None
dynamo_frontend_url: Optional[str] = None
+ dynamo_generation_cut_url: Optional[str] = None
@classmethod
def validate_settings(cls, master_config: "MasterConfig") -> None:
"""Reject pure-config vLLM settings the SC entrypoint cannot honor."""
generation_config = cast(VllmConfig, master_config.policy["generation"])
assert_reload_refit_config_supported(generation_config)
- if (
- dynamo_enabled(generation_config)
- and master_config.async_rl.generation_fleet_health.enabled
- ):
- raise ValueError(
- "async_rl.generation_fleet_health.enabled=true is not supported "
- "with policy.generation.dynamo=true yet: the Dynamo frontend "
- "picks engines itself, and a restarted shard does not register "
- "with it. Disable fleet health."
- )
+ if dynamo_enabled(generation_config):
+ if master_config.async_rl.generation_router.enabled:
+ raise ValueError(
+ "async_rl.generation_router.enabled=true does not apply with "
+ "policy.generation.dynamo=true: NeMo-Gym calls the Dynamo "
+ "frontend, which routes the whole fleet behind one URL. "
+ "Disable the generation router."
+ )
+ if master_config.async_rl.generation_fleet_health.enabled:
+ raise ValueError(
+ "async_rl.generation_fleet_health.enabled=true is not supported "
+ "with policy.generation.dynamo=true yet: the Dynamo frontend "
+ "picks engines itself, and a restarted shard does not register "
+ "with it. Disable fleet health."
+ )
+ if master_config.env.get("should_use_nemo_gym") and generation_config[
+ "vllm_cfg"
+ ].get("skip_tokenizer_init"):
+ raise ValueError(
+ "vllm_cfg.skip_tokenizer_init=true does not work with NeMo-Gym "
+ "and policy.generation.dynamo=true: the engine detokenizes the "
+ "replies Gym's chat calls get through the frontend."
+ )
@staticmethod
def init_cluster_placement_groups(
@@ -742,11 +758,24 @@ def load_and_start(self) -> None:
heavy model loading. Updates dp_openai_server_base_urls with the actual
running server URLs and populates device_uuids.
"""
+ chat_settings: Optional[RayQueue] = None
+ if self._dynamo_runtime is not None and self._dynamo_runtime.serves_chat:
+ # The frontend builds its render workers from one engine's chat
+ # settings while the engines load, so NeMo-Gym's first calls do
+ # not wait for them.
+ chat_settings = RayQueue(actor_options={"num_cpus": 0})
+ leader = self.worker_group.workers[
+ self.worker_group.get_dp_leader_worker_idx(0)
+ ]
+ ray.get(leader.announce_dynamo_chat_settings_to.remote(chat_settings))
+
# Call load_model() on all model-owner workers
futures = self.worker_group.run_all_workers_single_data(
"load_model",
run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"],
)
+ if chat_settings is not None:
+ self._publish_dynamo_chat_settings(chat_settings, futures)
ray.get(futures)
# Post-init (collective rpc functions needed for async engine)
@@ -759,6 +788,29 @@ def load_and_start(self) -> None:
self.device_uuids = self._report_device_id()
self._register_dynamo_engines()
+ @property
+ def client_base_urls(self) -> list[Optional[str]]:
+ """Where OpenAI clients such as NeMo-Gym send requests.
+
+ The Dynamo frontend when it serves this fleet (``policy.generation.dynamo``);
+ otherwise every data-parallel shard's own server, as
+ ``dp_openai_server_base_urls``. Known before a deferred model load.
+ """
+ if self.dynamo_frontend_url is not None:
+ return [self.dynamo_frontend_url]
+ return self.dp_openai_server_base_urls
+
+ @property
+ def generation_cut_control_urls(self) -> Optional[list[str]]:
+ """Where NeMo-Gym sends checkpoint generation cuts, one per ``client_base_urls`` entry.
+
+ None when each base URL takes its own cuts, as every shard's server
+ does. The Dynamo frontend takes them on a port of its own.
+ """
+ if self.dynamo_generation_cut_url is not None:
+ return [self.dynamo_generation_cut_url]
+ return None
+
def _start_dynamo_services(self) -> None:
"""Start the etcd, NATS and frontend this fleet's engines register with."""
from nemo_rl.models.generation.dynamo.managed_runtime import (
@@ -771,10 +823,38 @@ def _start_dynamo_services(self) -> None:
# The workers' venv carries ai-dynamo, so the frontend runs the same
# release as every engine.
frontend_python=self.worker_group.py_executable,
+ ray_address=ray.get_runtime_context().gcs_address,
+ # NeMo-Gym's chat calls need the flag, and the native worker starts
+ # its chat server on it.
+ serves_chat=bool(self.cfg["vllm_cfg"].get("expose_http_server")),
)
runtime.start()
self._dynamo_runtime = runtime
self.dynamo_frontend_url = runtime.frontend_url
+ self.dynamo_generation_cut_url = runtime.generation_cut_control_url
+
+ def _publish_dynamo_chat_settings(
+ self, chat_settings: RayQueue, loads: list[ray.ObjectRef]
+ ) -> None:
+ """Hand the frontend the chat settings an engine sends before it loads.
+
+ Returns without them once any of ``loads`` has ended: registration
+ then gives the frontend its settings, as without this.
+ """
+ assert self._dynamo_runtime is not None
+ try:
+ while True:
+ try:
+ settings = chat_settings.get(timeout=1)
+ except Empty:
+ ended, _ = ray.wait(loads, num_returns=1, timeout=0)
+ if ended:
+ return
+ continue
+ self._dynamo_runtime.publish_chat_settings(settings)
+ return
+ finally:
+ chat_settings.shutdown()
def _register_dynamo_engines(self) -> None:
"""Serve every shard's engine through Dynamo; wait until it routes to all."""
@@ -873,6 +953,20 @@ def setup_token_capture(
)
self._token_capture_media = agreed_media_columns(ray.get(futures))
self._token_capture_setup = setup
+ if self._dynamo_runtime is not None:
+ # NeMo-Gym's calls through Dynamo read their prefix in the frontend,
+ # which also takes Gym's generation cuts for the engines.
+ self._dynamo_runtime.publish_chat_capture(
+ dp_cfg=dp_cfg,
+ staging_partition=staging_partition,
+ capture_media=capture_media,
+ generation_cut_control_token=(
+ generation_cut_control_token
+ if generation_prefix_cuts_enabled
+ else None
+ ),
+ generation_cut_control_timeout_s=generation_cut_control_timeout_s,
+ )
return self._token_capture_media
def get_data_plane_snapshots(self) -> list[dict[str, Any]]:
diff --git a/nemo_rl/models/generation/vllm/vllm_worker.py b/nemo_rl/models/generation/vllm/vllm_worker.py
index 37b651478ba..b25dbd1ca0f 100644
--- a/nemo_rl/models/generation/vllm/vllm_worker.py
+++ b/nemo_rl/models/generation/vllm/vllm_worker.py
@@ -45,7 +45,6 @@
VLLM_SPARSE_REFIT_TRANSPORTS,
VllmConfig,
parse_nvfp4_pertoken_rollout,
- resolve_vllm_video_config,
validate_nvfp4_pertoken_model,
vllm_fp32_lm_head_enabled,
)
@@ -60,9 +59,7 @@
pad_and_align_routed_expert_indices,
validate_rollout_prompt,
)
-from nemo_rl.models.generation.vllm.video_utils import (
- register_torchcodec_vllm_video_loader,
-)
+from nemo_rl.models.generation.vllm.video_utils import register_vllm_video_loader
from nemo_rl.models.generation.vllm.worker_utils import (
find_tokenizer_required_architectures,
resolve_distributed_executor_backend,
@@ -819,12 +816,7 @@ def _load_model(self, bundle_indices, seed):
if logprobs_mode is not None:
llm_kwargs["logprobs_mode"] = logprobs_mode
- video_config = resolve_vllm_video_config(self.cfg)
- if video_config is not None:
- register_torchcodec_vllm_video_loader(
- sampling_style=video_config.sampling_style,
- temporal_patch_size=video_config.temporal_patch_size,
- )
+ register_vllm_video_loader(self.cfg)
_configure_nvfp4_pertoken_engine_kwargs(self.cfg, llm_kwargs)
_use_vllm_sampling_defaults(llm_kwargs)
llm_kwargs.setdefault("worker_extension_cls", default_worker_extension_cls)
diff --git a/nemo_rl/models/generation/vllm/vllm_worker_async.py b/nemo_rl/models/generation/vllm/vllm_worker_async.py
index 1ca0dede692..bcd4acf6e3c 100644
--- a/nemo_rl/models/generation/vllm/vllm_worker_async.py
+++ b/nemo_rl/models/generation/vllm/vllm_worker_async.py
@@ -15,30 +15,23 @@
import asyncio
import contextvars
import copy
-import functools
import gc
-import importlib.metadata
-import inspect
import logging
-import os
import secrets
-import sys
import threading
import time
import uuid
import warnings
-from collections.abc import Awaitable, Callable, Sequence
-from concurrent.futures import ThreadPoolExecutor
-from dataclasses import dataclass, field, replace
-from types import SimpleNamespace
-from typing import TYPE_CHECKING, Any, AsyncGenerator, Literal, Optional, cast
+from collections.abc import Awaitable, Callable
+from dataclasses import replace
+from typing import Any, AsyncGenerator, Optional, cast
import ray
import torch
import uvicorn
-from packaging.version import Version
from fastapi import FastAPI, Request, Response
from fastapi.responses import JSONResponse
+from ray.util.queue import Queue as RayQueue
from nemo_rl.data.captured_media import (
MEDIA_TENSOR_DIGESTS_FIELD,
@@ -65,36 +58,27 @@
from nemo_rl.distributed.worker_group_utils import get_nsight_config_if_pattern_matches
from nemo_rl.experience.route_assembly import boundary_route_rows
from nemo_rl.models.generation.dynamo.config import dynamo_cfg, dynamo_enabled
+from nemo_rl.models.generation.interfaces import (
+ GenerationDatumSpec,
+ GenerationOutputSpec,
+ verify_right_padding,
+)
+from nemo_rl.models.generation.openai_server_utils import (
+ PrefixSplice,
+)
from nemo_rl.models.generation.prefix_recovery import (
CaptureRegistry,
NoPrefixRecovery,
PrefixRecovery,
PrefixRecoveryParts,
)
-from nemo_rl.models.generation.prefix_recovery.restore import (
- _classify_restored_prefix_terminal,
- _full_generator_signature,
- _RestoredPrefixTerminal,
- mark_restored_terminal,
- plan_restore,
- restore_reply_usage,
- restored_terminal,
- wrap_parser_for_restore,
-)
from nemo_rl.models.generation.prefix_recovery.restore import (
_build_restored_prefix_terminal_output,
+ restored_terminal,
)
from nemo_rl.models.generation.prefix_recovery.state import _RequestCaptureState
-from nemo_rl.models.generation.interfaces import (
- GenerationDatumSpec,
- GenerationOutputSpec,
- verify_right_padding,
-)
from nemo_rl.models.generation.rollout_result import RolloutResult
from nemo_rl.models.generation.score_centering import validate_score_centering_request
-from nemo_rl.models.generation.vllm.captured_chat_logprobs import (
- CapturedChatLogprobsMixin,
-)
from nemo_rl.models.generation.vllm.checkpoint_engine import (
VllmAsyncCheckpointEngineRpcMixin,
)
@@ -106,30 +90,17 @@
parse_nvfp4_pertoken_rollout,
)
from nemo_rl.models.generation.vllm.multi_api import MultiAPIServerPool
+from nemo_rl.models.generation.vllm.progress_reader import VllmProgressReader
from nemo_rl.models.generation.vllm.request_pipeline import (
rollout_output,
rollout_result,
)
from nemo_rl.models.generation.vllm.utils import (
- attach_routed_experts_to_chat_response_choices,
- attach_token_information_to_chat_response_choices,
- attach_sampler_topk_logprobs_to_chat_response_choices,
format_prompt_for_vllm_generation,
- check_sampling_param,
+ take_reasoning_ended,
validate_rollout_prompt,
- model_dump_chat_response_with_dynamic_message_fields,
- remap_multimodal_placeholders,
)
from nemo_rl.models.generation.vllm.vllm_worker import BaseVllmGenerationWorker
-from nemo_rl.models.generation.vllm.generation_cut_route import (
- register_generation_cut_route,
-)
-from nemo_rl.models.generation.vllm.progress_reader import VllmProgressReader
-from nemo_rl.models.generation.capture_context import decide_capture_input
-from nemo_rl.models.generation.openai_server_utils import (
- PrefixSplice,
- splice_prefix_tokens,
-)
from nemo_rl.telemetry.setup import shutdown_telemetry
from nemo_rl.utils.routed_experts_codec import routed_experts_tensor_metadata
@@ -264,6 +235,7 @@ def generate(
and sampling_params.prompt_logprobs is None
):
sampling_params.flat_logprobs = True
+ take_reasoning_ended(sampling_params, kwargs)
return self._generate(prompt, sampling_params, request_id, kwargs)
async def _generate(
@@ -421,6 +393,8 @@ def __init__(
# The task serving this engine through Dynamo, when it is registered.
self._dynamo_endpoint_task: Optional[asyncio.Task] = None
+ # Where this engine sends its chat settings before it loads, when asked to.
+ self._dynamo_chat_settings_queue: Optional[RayQueue] = None
# Initialize HTTP server, token capture, and API server pool state.
self._init_frontend_state()
@@ -598,7 +572,7 @@ def _create_engine(self, llm_kwargs: dict[str, Any]) -> None:
)
if dynamo_enabled(self.cfg):
- from nemo_rl.models.generation.dynamo.vllm_adapter import (
+ from nemo_rl.models.generation.dynamo.vllm.adapter import (
SnapshotStatLogger,
dynamo_kv_events_config,
)
@@ -623,6 +597,15 @@ def _create_engine(self, llm_kwargs: dict[str, Any]) -> None:
)
self.llm_async_engine_args = AsyncEngineArgs(**llm_kwargs)
+ if self._dynamo_chat_settings_queue is not None:
+ # ai-dynamo is optional: import it only for engines served through Dynamo.
+ from nemo_rl.models.generation.dynamo.vllm.adapter import (
+ chat_runtime_data,
+ )
+
+ # Sent before the engine is built, which takes far longer than the
+ # frontend's render workers do.
+ self._dynamo_chat_settings_queue.put(chat_runtime_data(self))
# Initialize the vLLM engine
api_server_count = self.cfg["vllm_cfg"].get("http_api_server_count", 1)
@@ -663,7 +646,7 @@ def _start_vllm_metrics_logger(self) -> None:
Controlled by the required vllm_metrics_logger_interval in vllm_cfg.
Runs only on the model-owner actor.
"""
- from vllm.v1.metrics.reader import Gauge, Counter, get_metrics_snapshot
+ from vllm.v1.metrics.reader import Counter, Gauge, get_metrics_snapshot
assert self.cfg["vllm_cfg"].get("async_engine", False), (
"vLLM metrics logger is only supported with async engine enabled"
@@ -816,6 +799,15 @@ async def post_init_async(self):
if self._api_server_pool is not None:
await self._api_server_pool.command("start")
+ def announce_dynamo_chat_settings_to(self, queue: RayQueue) -> None:
+ """Have this engine send ``queue`` its chat settings before it loads.
+
+ It sends what it publishes at registration, as soon as it has its engine
+ args. Dynamo's frontend builds its render workers from them while the
+ engine loads.
+ """
+ self._dynamo_chat_settings_queue = queue
+
async def start_dynamo_endpoint_async(
self, *, namespace: str, engine_env: dict[str, str], leader_worker_idx: int
) -> None:
@@ -827,20 +819,15 @@ async def start_dynamo_endpoint_async(
engine serves report it, as ``generate_async`` results do.
"""
# ai-dynamo is optional: import it only for engines served through Dynamo.
- from dynamo._core import fetch_model
-
from nemo_rl.models.generation.dynamo.engine_adapter import (
start_dynamo_endpoint,
+ weight_free_model_path,
)
- from nemo_rl.models.generation.dynamo.vllm_adapter import DynamoVllmAdapter
+ from nemo_rl.models.generation.dynamo.vllm.adapter import DynamoVllmAdapter
# Dynamo fetches a Hub ID's whole checkpoint before it registers; the engine's
# weights come from refit, so register the config and tokenizer files only.
- model_path = (
- self.model_name
- if os.path.exists(self.model_name)
- else await fetch_model(self.model_name, ignore_weights=True)
- )
+ model_path = await weight_free_model_path(self.model_name)
kv_events = self.llm.vllm_config.kv_events_config
publishes = kv_events is not None and kv_events.enable_kv_cache_events
# The Dynamo functional overlays check the router applied these.
@@ -1079,7 +1066,10 @@ def _capture_admission(self, request: Any) -> Any | None:
context. The dict itself is never mutated: the admission is the typed,
read-only contract that the prefix resolution and ``begin_call`` share.
"""
- context = getattr(request, "ng_capture", None)
+ return self._parse_capture_admission(getattr(request, "ng_capture", None))
+
+ def _parse_capture_admission(self, context: Any) -> Any | None:
+ """``_capture_admission`` for a raw ``ng_capture`` context (Dynamo's annotation)."""
if self.token_capture is None or not context:
return None
# Deferred: nemo_gym is an optional extra absent in non-gym runs.
@@ -1100,8 +1090,8 @@ def _begin_request_capture(
) -> None:
"""Admit one ledger-forwarded call into the capture layer.
- Called from preprocess_chat once the exact engine prompt is known
- (post-splice in token-in mode, full render in text mode). No-op
+ Called from ``prompt_and_capture.start_capture`` once the exact engine prompt
+ is known (post-splice in token-in mode, full render in text mode). No-op
unless capture is installed and the request carries the ledger's
``ng_capture`` context.
@@ -1316,7 +1306,7 @@ def _capture_restored_media(
engine_prompt: dict[str, Any],
*,
admission: Any,
- splice: PrefixSplice,
+ splice: PrefixSplice | None,
snapshot: Any,
) -> CapturedMedia | None:
"""Reprocess a restored call's media and check it against the digests its cut saved."""
@@ -1340,18 +1330,6 @@ def _resolve_admission_prefix(self, admission: Any) -> list[int]:
return resolve_admission_prefix(admission, self._chain_prefix)
- def _enter_request_prefix(self, request: Any, prefix_token_ids: list[int]) -> None:
- """Attach the resolved prefix to the request through the capture adapter.
-
- ``VLLMCaptureAdapter.enter_prefix`` writes the engine-native field
- (``required_prefix_token_ids``) into a payload; the same fields are
- applied to the pydantic request so the existing prefix-splice branch
- of preprocess_chat handles staged and inline prefixes alike.
- """
- adapter = self.token_capture.adapter
- for field_name, value in adapter.enter_prefix({}, prefix_token_ids).items():
- setattr(request, field_name, value)
-
@staticmethod
def _delta_align_routed_experts(
payload: dict[str, Any], *, prev_len: int, prompt_len: int, generated_len: int
@@ -1570,1037 +1548,21 @@ def _context_refusal_headers(self, request: Any) -> dict[str, str]:
return {CAPTURE_REFUSAL_HEADER: admission.model_call_id}
- # ruff: noqa
def _setup_vllm_openai_api_server(self, app: FastAPI) -> FastAPI:
- worker_self = self
- from copy import deepcopy
- from logging import Filter as LoggingFilter
- from logging import LogRecord
- from typing import List, Optional, Union
-
- from fastapi import Header, HTTPException, Request
- from fastapi.responses import JSONResponse, StreamingResponse
- from pydantic import PrivateAttr
- from vllm.renderers.chat_utils.chat_template import load_chat_template
- from vllm.entrypoints.openai.chat_completion.protocol import (
- ChatCompletionRequest,
- ChatCompletionResponse,
- )
- from vllm.entrypoints.openai.chat_completion.serving import (
- OpenAIServingChat,
- )
-
- full_generator_signature = _full_generator_signature(
- OpenAIServingChat.chat_completion_full_generator
- )
-
- # vLLM 0.29 moved this out of the openai package (vllm-project/vllm#54492).
- from vllm.entrypoints.serve.engine.protocol import ErrorResponse
- from vllm.entrypoints.openai.models.protocol import BaseModelPath
- from vllm.entrypoints.openai.models.serving import OpenAIServingModels
- from vllm.entrypoints.serve.tokenize.protocol import (
- TokenizeChatRequest,
- TokenizeCompletionRequest,
- TokenizeResponse,
- )
- from vllm.entrypoints.serve.tokenize.serving import (
- ServingTokenization,
- )
- from vllm.entrypoints.serve.utils.api_utils import with_cancellation
- from vllm.renderers.online_renderer import OnlineRenderer
- from vllm.renderers.hf import HfRenderer
- from vllm.exceptions import VLLMValidationError
- from vllm.reasoning.abs_reasoning_parsers import ReasoningParserManager
- from vllm.tool_parsers.abstract_tool_parser import ToolParserManager
- from vllm.v1.engine.async_llm import logger as vllm_async_llm_logger
-
- maybe_tool_parser_plugin = self.cfg["vllm_cfg"].get("tool_parser_plugin")
- if maybe_tool_parser_plugin:
- ToolParserManager.import_tool_parser(maybe_tool_parser_plugin)
-
- maybe_reasoning_parser_plugin = self.cfg["vllm_cfg"].get(
- "reasoning_parser_plugin"
- )
- if maybe_reasoning_parser_plugin:
- ReasoningParserManager.import_reasoning_parser(
- maybe_reasoning_parser_plugin
- )
+ # Deferred: the chat server imports vLLM's OpenAI server stack.
+ from nemo_rl.models.generation.vllm import chat_server
engine_client = self._http_engine_client
if engine_client is None:
raise RuntimeError("The HTTP engine client is not initialized.")
- model_config = self.llm_async_engine_args.create_model_config()
- # vLLM adds the generation config's EOS to every request's stop set
- # regardless of the configured stops, so a turn can end on either.
- model_stop_token_ids = set(self.cfg.get("stop_token_ids") or ())
- generation_eos_token_ids = model_config.try_get_generation_config().get(
- "eos_token_id"
- )
- if isinstance(generation_eos_token_ids, int):
- generation_eos_token_ids = [generation_eos_token_ids]
- model_stop_token_ids |= set(generation_eos_token_ids or ())
- base_model_paths = [
- BaseModelPath(
- name=model_config.served_model_name, model_path=model_config.model
- ),
- BaseModelPath(name=model_config.model, model_path=model_config.model),
- ]
-
- openai_serving_models_kwargs = dict(
+ server = chat_server.build_chat_server(
+ self,
engine_client=engine_client,
- base_model_paths=base_model_paths,
- lora_modules=None,
- )
- openai_serving_models = OpenAIServingModels(**openai_serving_models_kwargs)
-
- class NeMoRLOpenAIChatRequestMixin:
- def model_post_init(self, context):
- # NeMo-Gym specific processing. This is just how NeMo-Gym returns the extra token information.
- if self.required_prefix_token_ids is None:
- for message in reversed(self.messages):
- if "prompt_token_ids" in message:
- self.required_prefix_token_ids = (
- message["prompt_token_ids"]
- + message["generation_token_ids"]
- )
- break
-
- return super().model_post_init(context)
-
- class NeMoRLOpenAIServingMixin:
- @staticmethod
- def _set_max_tokens(request, max_tokens: int) -> None:
- """Set the request's max output tokens.
-
- Mutates the request in place. Handles both max_completion_tokens (newer OpenAI API)
- and max_tokens (deprecated but still supported by vLLM). A request
- that set neither gets max_tokens: otherwise vLLM would fall back to
- the generation config's budget, e.g. for a restored call whose
- remaining budget is smaller.
- """
- if request.max_completion_tokens is not None:
- request.max_completion_tokens = max_tokens
- else:
- request.max_tokens = max_tokens
-
- def _clamp_max_tokens(
- self, request, request_max_tokens: int, prompt_token_ids: list[int]
- ) -> None:
- """Clamp the request's max output tokens so that input + output <= max_model_len."""
- remaining = self.model_config.max_model_len - len(prompt_token_ids)
- if remaining <= 0:
- # preserve the literal "context length" in this message to match Gym's overflow handling
- message = (
- f"Prompt length ({len(prompt_token_ids)}) fills or exceeds "
- f"this model's maximum context length ({self.model_config.max_model_len}). "
- f"No room for output tokens."
- )
- LOGGER.warning("Prompt exceeds max_model_len: %s", message)
- raise VLLMValidationError(
- message,
- parameter="input_tokens",
- value=len(prompt_token_ids),
- )
- max_tokens = min(request_max_tokens, remaining)
- self._set_max_tokens(request, max_tokens)
-
- # vLLM 0.25 moved chat preprocessing to
- # OnlineRenderer.preprocess_chat (tool_parser/reasoning_parser were
- # folded into a single `parser`), so this override now applies via
- # the renderer subclass.
- async def preprocess_chat(
- self,
- request,
- messages,
- default_template,
- default_template_content_format,
- default_template_kwargs,
- tool_dicts=None,
- parser=None,
- *,
- skip_mm_cache: bool = False,
- ):
- for message in messages:
- if message.get("tool_calls"):
- message["tool_calls"] = list(message["tool_calls"])
-
- messages_for_replace_prefix_tokens = deepcopy(messages)
- # #4124: processor-only cache reads retain concrete pixels even
- # when the engine's sender cache would return references.
- if (
- worker_self._capture_media
- and worker_self._capture_admission(request) is not None
- ):
- skip_mm_cache = True
-
- # Temporarily set to 1 so vLLM's pre-tokenization length check passes;
- # the actual value will be set through _clamp_max_tokens later.
- actual_request_max_tokens = None
- if isinstance(request, NeMoRLChatCompletionRequest):
- actual_request_max_tokens = (
- request.max_completion_tokens
- if request.max_completion_tokens is not None
- else request.max_tokens
- )
- # If max_completion_tokens or max_tokens is not set, we don't need to do _clamp_max_tokens.
- # So we don't need to set the request's max output tokens to 1 here.
- if actual_request_max_tokens is not None:
- self._set_max_tokens(request, 1)
-
- # vLLM 0.33 turns parallel_tool_calls=False into a grammar that
- # masks sampling after the first tool call, so the logprobs it
- # returns no longer match the trainer's unmasked forward pass.
- # Hide the flag from the parser only: vLLM still trims the
- # response to the first tool call, as it did before 0.33.
- single_tool_call = (
- getattr(request, "parallel_tool_calls", None) is False
- )
- if single_tool_call:
- request.parallel_tool_calls = None
- try:
- res = await super().preprocess_chat(
- request=request,
- messages=messages,
- default_template=default_template,
- default_template_content_format=default_template_content_format,
- default_template_kwargs=default_template_kwargs,
- tool_dicts=tool_dicts,
- parser=parser,
- skip_mm_cache=skip_mm_cache,
- )
- except (ValueError, VLLMValidationError) as e:
- if "maximum context length" in str(e):
- import logging
-
- logging.getLogger(__name__).warning(
- "Prompt exceeds max_model_len: %s", e
- )
- raise
- finally:
- if single_tool_call:
- request.parallel_tool_calls = False
-
- # Token capture: build the admission once, before branching,
- # and resolve its prefix from it (staging_chain -> cached TQ
- # read, inline ids, or nothing for a text root). The
- # ``ng_capture`` dict is never mutated. Off-loop: the chain
- # fetch is a blocking TQ read, and Gym's staging protocol
- # requires the serving host to move blocking staging I/O off
- # its event loop explicitly. The adapter then attaches the
- # prefix to the request, so the inline-prefix branch below is
- # the single splice path for staged and inline prefixes.
- admission = worker_self._capture_admission(request)
- serving_admission = admission
- verify_retained_media = False
- if admission is not None and (
- admission.mode == "candidate"
- or admission.request_replay is not None
- ):
- # Inline assistant token fields cannot override RL's decision.
- request.required_prefix_token_ids = None
- decision = decide_capture_input(
- admission,
- messages=messages_for_replace_prefix_tokens,
- # HF/string parsing maps null and empty tool content to
- # identical template inputs. Structured content and
- # other renderers cannot authorize that equivalence.
- allow_empty_tool_content=(
- type(self.renderer) is HfRenderer
- and bool(res[0])
- and all(
- isinstance(item.get("content"), str) for item in res[0]
- )
- ),
- )
- admission = decision.storage
- serving_admission = decision.serving
- verify_retained_media = decision.verify_retained_media
- capture_prefix_token_ids: list[int] | None = None
- generation_cut = None
- resumed_generation_token_ids: list[int] = []
- restored_request_output_tokens: int | None = None
- if serving_admission is not None:
- capture_prefix_token_ids = await asyncio.to_thread(
- worker_self._resolve_admission_prefix, serving_admission
- )
- resolved_admission, generation_cut = await asyncio.to_thread(
- worker_self._prefix_recovery.resolve_restore,
- serving_admission,
- capture_prefix_token_ids,
- )
- # An unusable prefix comes back without its cut, so the
- # call regenerates; storage and serving share one admission here.
- if serving_admission is admission:
- admission = resolved_admission
- serving_admission = resolved_admission
- engine_prefix_token_ids = list(capture_prefix_token_ids)
- if generation_cut is not None:
- plan = plan_restore(
- prefix_token_ids=capture_prefix_token_ids,
- saved=generation_cut,
- continuation=serving_admission.generation_cut,
- min_tokens=getattr(request, "min_tokens", None),
- )
- restored_request_output_tokens = plan.output_limit
- engine_prefix_token_ids = plan.engine_prefix_token_ids
- resumed_generation_token_ids = list(
- plan.resumed_generation_token_ids
- )
- request._restored_generation_token_ids = (
- plan.resumed_generation_token_ids
- )
- request._original_min_tokens = getattr(
- request, "min_tokens", None
- )
- if plan.max_tokens is not None:
- actual_request_max_tokens = plan.max_tokens
- if plan.min_tokens is not None:
- request.min_tokens = plan.min_tokens
- if engine_prefix_token_ids:
- worker_self._enter_request_prefix(
- request, engine_prefix_token_ids
- )
-
- if (
- not hasattr(request, "required_prefix_token_ids")
- or request.required_prefix_token_ids is None
- ):
- # Clamp the request's max output tokens so that input + output <= max_model_len.
- if actual_request_max_tokens is not None:
- self._clamp_max_tokens(
- request,
- actual_request_max_tokens,
- res[1][0]["prompt_token_ids"],
- )
- # Token capture, text mode: the full render is the exact
- # engine prompt.
- media = await asyncio.to_thread(
- worker_self._capture_request_media,
- res[1][0],
- admission=admission,
- verify_retained_media=verify_retained_media,
- )
- worker_self._begin_request_capture(
- request,
- res[1][0]["prompt_token_ids"],
- admission=admission,
- prefix_token_ids=capture_prefix_token_ids,
- media=media,
- generation_cut=generation_cut,
- resumed_generation_token_ids=resumed_generation_token_ids,
- )
- return res
-
- model_prefix_token_ids = list(request.required_prefix_token_ids)
-
- # Token-in splice path — shared by staging_chain and direct prefix.
- last_assistant_message_idx = None
- for i in reversed(range(len(messages_for_replace_prefix_tokens))):
- if messages_for_replace_prefix_tokens[i]["role"] == "assistant":
- last_assistant_message_idx = i
- break
-
- if last_assistant_message_idx is None:
- messages_to_last_assistant_message = (
- messages_for_replace_prefix_tokens
- )
- else:
- messages_to_last_assistant_message = (
- messages_for_replace_prefix_tokens[
- : last_assistant_message_idx + 1
- ]
- )
-
- modified_request = request.model_copy(
- update={"add_generation_prompt": False}
- )
-
- corresponding_res = await super().preprocess_chat(
- request=modified_request,
- messages=messages_to_last_assistant_message,
- default_template=default_template,
- default_template_content_format=default_template_content_format,
- default_template_kwargs=default_template_kwargs,
- tool_dicts=tool_dicts,
- parser=parser,
- skip_mm_cache=skip_mm_cache,
- )
- actual_corresponding_token_ids = corresponding_res[1][0][
- "prompt_token_ids"
- ]
-
- engine_prompt = res[1][0]
-
- splice = None
- if generation_cut is not None:
- # The durable prefix is the exact engine prompt at the cut;
- # rendering/splicing it again could add or drop tokens.
- # truncate_prompt_tokens already applied when the original
- # call's chat was tokenized, and vLLM does not truncate
- # this list again (test_vllm_progress_read_layout.py).
- final_prompt_token_ids = model_prefix_token_ids
- else:
- splice = splice_prefix_tokens(
- tokenizer=self.renderer.tokenizer,
- model_prefix_token_ids=model_prefix_token_ids,
- template_prefix_token_ids=actual_corresponding_token_ids,
- template_token_ids=engine_prompt["prompt_token_ids"],
- # The template's own EOS must stay countable even when
- # the configured stops don't include it.
- eos_token_id=model_stop_token_ids
- | {self.renderer.tokenizer.eos_token_id},
- )
- final_prompt_token_ids = splice.token_ids
-
- restored_prefix_terminal = None
- if generation_cut is not None:
- try:
- restored_prefix_terminal = _classify_restored_prefix_terminal(
- prompt_token_ids=final_prompt_token_ids,
- prefix_token_count=(
- admission.generation_cut.prefix_token_count
- ),
- requested_output_tokens=restored_request_output_tokens,
- model_max_tokens=self.model_config.max_model_len,
- terminal_finish_reason=(
- admission.generation_cut.terminal_finish_reason
- ),
- terminal_stop_reason=(
- admission.generation_cut.terminal_stop_reason
- ),
- )
- except ValueError as error:
- raise VLLMValidationError(
- str(error),
- parameter="generation_prefix",
- value=admission.generation_cut.prefix_token_count,
- ) from error
-
- # Reject overflows before media validation can mask the length
- # error. A restored terminal prefix generates nothing more.
- if (
- restored_prefix_terminal is None
- and actual_request_max_tokens is not None
- ):
- self._clamp_max_tokens(
- request,
- actual_request_max_tokens,
- final_prompt_token_ids,
- )
-
- media = None
- if generation_cut is None:
- media = await asyncio.to_thread(
- worker_self._capture_request_media,
- engine_prompt,
- admission=serving_admission,
- splice=splice,
- verify_retained_media=verify_retained_media,
- )
- elif worker_self._capture_media:
- # A restored prompt is the durable prefix itself, so splice
- # the template only to locate its media placeholders.
- media_splice = splice_prefix_tokens(
- tokenizer=self.renderer.tokenizer,
- model_prefix_token_ids=list(capture_prefix_token_ids or ()),
- template_prefix_token_ids=actual_corresponding_token_ids,
- template_token_ids=engine_prompt["prompt_token_ids"],
- )
- media = await asyncio.to_thread(
- worker_self._capture_restored_media,
- engine_prompt,
- admission=admission,
- splice=media_splice,
- snapshot=generation_cut,
- )
- # Must precede the reassignment below: the remap reads the
- # current value of prompt_token_ids as the template-coordinate
- # sequence the placeholders were computed against. Reordering
- # these two statements silently degrades to a no-op.
- # Successful media capture already remapped these ranges.
- if media is None and (
- mm_placeholders := engine_prompt.get("mm_placeholders")
- ):
- engine_prompt["mm_placeholders"] = remap_multimodal_placeholders(
- template_token_ids=engine_prompt["prompt_token_ids"],
- final_token_ids=final_prompt_token_ids,
- mm_placeholders=mm_placeholders,
- splice=splice,
- )
-
- engine_prompt["prompt_token_ids"] = final_prompt_token_ids
-
- # Token capture, token-in mode: the spliced prompt is the
- # exact engine prompt; begin_call re-checks the prefix it
- # was spliced from against the admission.
- worker_self._begin_request_capture(
- request,
- final_prompt_token_ids,
- admission=admission,
- prefix_token_ids=(
- capture_prefix_token_ids
- if admission is not None and admission.mode == "token_in"
- else None
- ),
- media=media,
- generation_cut=generation_cut,
- resumed_generation_token_ids=resumed_generation_token_ids,
- )
-
- if restored_prefix_terminal is not None:
- # The engine client answers it without decoding; one
- # token is the smallest budget vLLM accepts, so a missed
- # marker costs one token, not the whole call.
- request.min_tokens = 0
- self._set_max_tokens(request, 1)
- mark_restored_terminal(restored_prefix_terminal)
- if len(final_prompt_token_ids) == self.model_config.max_model_len:
- engine_prompt["prompt_token_ids"] = final_prompt_token_ids[:-1]
- request._restored_prefix_terminal = restored_prefix_terminal
- LOGGER.info(
- "generation prefix already terminal: "
- "rollout_id=%s model_call_id=%s prefix_tokens=%d reason=%s",
- admission.capture_key,
- admission.model_call_id,
- restored_prefix_terminal.prefix_token_count,
- restored_prefix_terminal.reason,
- )
-
- return res
-
- ########################################
- # /v1/chat/completions endpoint
- ########################################
-
- # This MRO is necessary i.e. NeMoRLOpenAIChatRequestMixin > ChatCompletionRequest
- class NeMoRLChatCompletionRequest(
- NeMoRLOpenAIChatRequestMixin, ChatCompletionRequest
- ):
- required_prefix_token_ids: Optional[List[int]] = None
- # Ledger-authoritative token capture: the call identity the ledger
- # attaches (capture_key, call_id, parent_call_id, prev_len, mode).
- ng_capture: Optional[dict[str, Any]] = None
- _restored_prefix_terminal: _RestoredPrefixTerminal | None = PrivateAttr(
- default=None
- )
- _restored_generation_token_ids: tuple[int, ...] = PrivateAttr(default=())
- # The client's min_tokens before it is reduced by a restored prefix;
- # replaying the prefix's stop handling needs the original window.
- _original_min_tokens: int | None = PrivateAttr(default=None)
-
- def to_sampling_params(self, *args, **kwargs):
- sampling_params = super().to_sampling_params(*args, **kwargs)
- if self.ng_capture is not None:
- worker_self._prefix_recovery.on_sampling_params(
- self, sampling_params
- )
- return sampling_params
-
- # vLLM 0.25 routes both /v1/chat/completions and /tokenize through
- # OnlineRenderer.preprocess_chat, so the prefix-token override
- # belongs on the renderer subclass.
- worker_self = self
-
- register_generation_cut_route(app, lambda: worker_self._prefix_recovery)
-
- class NeMoRLOpenAIServingChatMixin:
- async def chat_completion_full_generator(
- self,
- request,
- result_generator,
- *args,
- **kwargs,
- ):
- return_as_token_id = (
- request.return_tokens_as_token_ids
- if request.return_tokens_as_token_ids is not None
- else self.return_tokens_as_token_ids
- )
- if (
- request.logprobs
- and return_as_token_id
- and request.top_logprobs is None
- ):
- raise VLLMValidationError(
- "`top_logprobs` must be set when requesting token "
- "information from the NeMo-RL chat endpoint.",
- parameter="top_logprobs",
- )
-
- final_res = None
- capture_routes = request in worker_self._capture_registry
- cancelled = False
-
- def _without_routes(res):
- # vLLM encodes choice-level routes in its response. Keep
- # engine arrays for our native capture, and hide them only
- # in the superclass's shallow view.
- res = copy.copy(res)
- res.outputs = [copy.copy(output) for output in res.outputs]
- for output in res.outputs:
- output.routed_experts = None
- return res
-
- async def capture_result_generator():
- nonlocal final_res, cancelled
- try:
- async for res in result_generator:
- # Recorded before asking for the next output, which
- # releases vLLM's finished state for this request.
- worker_self._prefix_recovery.observe(request, res)
- final_res = res
- if final_res is None:
- return
- worker_self._prefix_recovery.restore_reply_text(
- request,
- final_res,
- tokenizer=self.renderer.tokenizer,
- )
- yield (
- _without_routes(final_res) if capture_routes else final_res
- )
- except asyncio.CancelledError:
- cancelled = True
- raise
-
- # Bound by name: vLLM passes request_metadata by position and
- # parser by keyword, and either may move.
- bound = full_generator_signature.bind_partial(
- self, request, capture_result_generator(), *args, **kwargs
- )
- if bound.arguments.get("parser") is not None:
- bound.arguments["parser"] = wrap_parser_for_restore(
- bound.arguments["parser"],
- request._restored_generation_token_ids,
- )
-
- # Capture stages the raw arrays below and strips OpenAI logprobs
- # after the terminal PUT. Skip only their object construction;
- # changing request.logprobs would also change engine sampling.
- bypass_logprobs = bool(
- not request.stream
- and request.logprobs
- and return_as_token_id
- and request in worker_self._capture_registry
- )
- with self._bypass_chat_logprobs(enabled=bypass_logprobs):
- response = await super().chat_completion_full_generator(
- *bound.args[1:], **bound.kwargs
- )
- if cancelled:
- # vLLM's chat_completion_full_generator catches the
- # CancelledError that with_cancellation throws into the
- # handler task on a client disconnect and returns
- # ErrorResponse("Client disconnected") instead of
- # re-raising. Nobody reads that response (the wrapper has
- # already returned), and letting it through would make the
- # route label the capture "error_response". Re-raise so the
- # task finishes cancelled and the route sees the disconnect.
- raise asyncio.CancelledError()
- if (
- not isinstance(response, ChatCompletionResponse)
- or final_res is None
- ):
- return response
-
- restored_prefix_terminal = request._restored_prefix_terminal
- if not isinstance(restored_prefix_terminal, _RestoredPrefixTerminal):
- restored_prefix_terminal = None
- if restore_reply_usage(
- response,
- restored_token_count=len(request._restored_generation_token_ids),
- terminal=restored_prefix_terminal,
- ):
- request_metadata = bound.arguments.get("request_metadata")
- if request_metadata is not None:
- request_metadata.final_usage_info = response.usage
-
- if request.logprobs and return_as_token_id:
- response = attach_token_information_to_chat_response_choices(
- response,
- final_res,
- )
- if request.ng_capture and worker_self._capture_top_logprobs:
- # Extraction scales with completion length and K.
- response = await asyncio.to_thread(
- attach_sampler_topk_logprobs_to_chat_response_choices,
- response,
- final_res,
- top_k=worker_self._capture_top_logprobs,
- )
-
- if worker_self._return_routed_experts_enabled():
- response = attach_routed_experts_to_chat_response_choices(
- response,
- final_res,
- device=torch.device("cpu"),
- logger=LOGGER,
- routed_experts_dtype=worker_self.routed_experts_dtype,
- encode_for_wire=not capture_routes,
- )
-
- return response
-
- class NeMoRLOpenAIServingChat(
- NeMoRLOpenAIServingChatMixin,
- CapturedChatLogprobsMixin,
- OpenAIServingChat,
- ):
- pass
-
- class NeMoRLOnlineRenderer(NeMoRLOpenAIServingMixin, OnlineRenderer):
- pass
-
- serving_chat_default_kwargs = dict(
- response_role="assistant",
- request_logger=None,
- chat_template=None,
- chat_template_content_format="auto",
- enable_auto_tools=True,
- )
- serving_chat_kwargs = serving_chat_default_kwargs | self.cfg["vllm_cfg"].get(
- "http_server_serving_chat_kwargs", dict()
- )
- # The embedded server is constructed directly instead of through
- # vLLM's CLI, where chat-template file paths are normally loaded.
- # OnlineRenderer expects literal Jinja content; passing a path makes
- # Transformers render the path itself and drops multimodal
- # placeholders such as .
- configured_chat_template = serving_chat_kwargs.get("chat_template")
- if configured_chat_template is not None:
- serving_chat_kwargs["chat_template"] = load_chat_template(
- configured_chat_template
- )
- # Recipes may name the parameter either way: ``default_chat_template_kwargs``
- # is vLLM's own spelling, ``chat_template_kwargs`` is accepted for recipes
- # written against the older name. Normalize onto the native key rather
- # than popping it: OnlineRenderer, OpenAIServingChat and ServingTokenization
- # each keep their *own* copy and read it independently -- the chat serving
- # builds its reasoning parser from it, and the tokenize path passes its own
- # into preprocess_chat -- so the renderer's copy does not reach either.
- # vLLM's api_server hands the same value to all three for that reason.
- #
- # Popped separately, not `A or B`: short-circuiting on a truthy A would
- # leave B in the bag and OpenAIServingChat(**kwargs) would reject it.
- _legacy_chat_template_kwargs = serving_chat_kwargs.pop(
- "chat_template_kwargs", None
- )
- if serving_chat_kwargs.get("default_chat_template_kwargs") is None:
- serving_chat_kwargs["default_chat_template_kwargs"] = (
- _legacy_chat_template_kwargs
- )
- default_chat_template_kwargs: dict[str, Any] = (
- serving_chat_kwargs["default_chat_template_kwargs"] or {}
- )
- online_renderer = NeMoRLOnlineRenderer(
- model_config=engine_client.model_config,
- renderer=engine_client.renderer,
- request_logger=serving_chat_kwargs["request_logger"],
- chat_template=serving_chat_kwargs["chat_template"],
- chat_template_content_format=serving_chat_kwargs[
- "chat_template_content_format"
- ],
- enable_auto_tools=serving_chat_kwargs["enable_auto_tools"],
- # Keep the renderer's parser consistent with any parser overrides
- # passed to OpenAIServingChat via http_server_serving_chat_kwargs.
- tool_parser=serving_chat_kwargs.get("tool_parser"),
- reasoning_parser=serving_chat_kwargs.get("reasoning_parser"),
- # vLLM merges these into every render, with request-supplied keys
- # winning (preprocess_chat's default_template_kwargs). The renderer
- # is shared by /v1/chat/completions and /tokenize, so setting it
- # here keeps the two endpoints rendering identical prompts.
- default_chat_template_kwargs=default_chat_template_kwargs,
- )
- serving_chat_kwargs.update(
- dict(
- engine_client=engine_client,
- models=openai_serving_models,
- online_renderer=online_renderer,
- return_tokens_as_token_ids=True,
- )
- )
- openai_serving_chat = NeMoRLOpenAIServingChat(**serving_chat_kwargs)
-
- generation_config = self.cfg
-
- async def abort_request_capture(request: Any, reason: str) -> None:
- """Fail a request's capture without blocking the event loop.
-
- An uncontended abort finishes here, so it never queues behind
- executor work. Only one that must wait out a generation cut or a
- terminal write holding the call's lifecycle lock moves off the loop.
- """
- if worker_self._abort_request_capture(
- request, reason=reason, blocking=False
- ):
- return
- # Shielded: these paths often run after the handler was cancelled,
- # and a second cancel must not drop the abort before it starts,
- # leaving the call neither completed nor failed.
- await asyncio.shield(
- asyncio.to_thread(
- worker_self._abort_request_capture, request, reason=reason
- )
- )
-
- # The create_chat_completion and tokenize methods are taken from vllm/entrypoints/openai/api_server.py
- @app.post("/v1/chat/completions")
- # Like vLLM's own routes: cancel the handler (and with it the engine
- # request) when the client disconnects, instead of generating for nobody.
- @with_cancellation
- async def create_chat_completion(
- request: NeMoRLChatCompletionRequest, raw_request: Request
- ):
- # Accept train and validation profiles; reject off-policy requests.
- # vLLM fills an unset top_k/min_p/repetition_penalty from
- # generation_config.json when vllm_kwargs sets generation_config="auto",
- # so pin top_k to the matched profile's (as
- # vllm_worker.py::BaseVllmGenerationWorker::_build_sampling_params does)
- # and disable min_p and the repetition penalty. vLLM applies these and
- # the presence/frequency penalties and logit bias before sampling; the
- # trainer replays none of them, so a request may not set them.
- top_k = check_sampling_param(
- temperature=request.temperature,
- top_p=request.top_p,
- top_k=request.top_k,
- generation_config=generation_config,
- )
- assert request.min_p in (None, 0.0), (
- f"request min_p={request.min_p} is not supported; leave it unset"
- )
- assert request.repetition_penalty in (None, 1.0), (
- f"request repetition_penalty={request.repetition_penalty} is not "
- "supported; leave it unset"
- )
- assert request.presence_penalty in (None, 0.0), (
- f"request presence_penalty={request.presence_penalty} is not "
- "supported; leave it unset"
- )
- assert request.frequency_penalty in (None, 0.0), (
- f"request frequency_penalty={request.frequency_penalty} is not "
- "supported; leave it unset"
- )
- assert not request.logit_bias, "request logit_bias is not supported"
- request.top_k = top_k
- request.min_p = 0.0
- request.repetition_penalty = 1.0
-
- try:
- generator = await openai_serving_chat.create_chat_completion(
- request, raw_request
- )
- except VLLMValidationError as e:
- # vLLM raises VLLMValidationError for prompts exceeding
- # max_model_len during tokenization, instead of returning an
- # ErrorResponse. Convert to HTTP 400 so the Gym proxy can
- # detect context-length overflow and handle it gracefully.
- await abort_request_capture(request, "context_length")
- return JSONResponse(
- content={
- "error": {
- "message": str(e),
- "type": "invalid_request_error",
- "param": e.parameter,
- "code": 400,
- }
- },
- status_code=400,
- headers=worker_self._context_refusal_headers(request),
- )
- except MediaCaptureRejected as e:
- # Raised inside preprocess_chat before begin_call, so no capture
- # state exists yet and the abort below is a no-op kept for
- # symmetry. Return a 400 carrying a stable code so Gym and the
- # worker log can distinguish a retained-media re-tile from a
- # real engine error (which stays a 500 below).
- await abort_request_capture(request, e.code)
- LOGGER.warning(
- "Rejected captured call before inference (%s): %s", e.code, e
- )
- return JSONResponse(
- content={
- "error": {
- "message": str(e),
- "type": "invalid_request_error",
- "param": "messages",
- "code": e.code,
- }
- },
- status_code=400,
- )
- except ValueError as e:
- # vLLM's get_max_tokens raises a plain ValueError ("Input length
- # (N) exceeds model's maximum context length (M).") after
- # preprocess_chat, e.g. when a token-in splice grows the prompt
- # past max_model_len on a request with no max_tokens to clamp.
- # Convert only that overflow to the same 400 as above; any other
- # ValueError is a server bug and stays a 500. Must follow
- # MediaCaptureRejected, which is also a ValueError.
- if "maximum context length" not in str(e):
- # Re-raising here bypasses the BaseException handler below.
- await abort_request_capture(request, "engine_error")
- raise
- LOGGER.warning("Prompt exceeds max_model_len: %s", e)
- await abort_request_capture(request, "context_length")
- return JSONResponse(
- content={
- "error": {
- "message": str(e),
- "type": "invalid_request_error",
- "param": "input_tokens",
- "code": 400,
- }
- },
- status_code=400,
- # Only vLLM's known pre-generation get_max_tokens refusal
- # can settle capture; an arbitrary ValueError cannot.
- headers=(
- worker_self._context_refusal_headers(request)
- if str(e).startswith("Input length (")
- and "exceeds model's maximum context length" in str(e)
- else None
- ),
- )
- except asyncio.CancelledError:
- # Client disconnected (see with_cancellation above). A cancel
- # before generation starts propagates here directly; one that
- # lands mid-generation is swallowed by vLLM and re-raised by
- # NeMoRLOpenAIServingChatMixin.chat_completion_full_generator.
- await abort_request_capture(request, "client_disconnect")
- raise
- except BaseException:
- await abort_request_capture(request, "engine_error")
- raise
-
- if isinstance(generator, ErrorResponse):
- await abort_request_capture(request, "error_response")
- return JSONResponse(
- content=generator.model_dump(), status_code=generator.error.code
- )
-
- elif isinstance(generator, ChatCompletionResponse):
- content = model_dump_chat_response_with_dynamic_message_fields(
- generator
- )
- # Token capture: stage the delta and ride the coords on the
- # response; strips logprobs/ids (no-op when capture is off).
- # Off-loop: the sink write inside complete_call is a blocking
- # TQ round trip (see the staging protocol's serving-host rule).
- try:
- content = await asyncio.to_thread(
- worker_self._finish_request_capture, request, content
- )
- except asyncio.CancelledError:
- # Queued executor work may never start after cancellation.
- # If staging already claimed the capture, abort is a no-op.
- await abort_request_capture(request, "client_disconnect")
- raise
- return JSONResponse(content=content)
-
- await abort_request_capture(request, "streaming_response")
- return StreamingResponse(content=generator, media_type="text/event-stream")
-
- ########################################
- # /tokenize endpoint
- ########################################
-
- # This MRO is necessary i.e. NeMoRLOpenAIChatRequestMixin > TokenizeRequest
- class NeMoRLTokenizeChatRequest(
- NeMoRLOpenAIChatRequestMixin, TokenizeChatRequest
- ):
- required_prefix_token_ids: Optional[List[int]] = None
- ng_capture: Optional[dict[str, Any]] = None
- ng_measure_only: Literal[True] = True
-
- NeMoRLTokenizeRequest = Union[
- TokenizeCompletionRequest, NeMoRLTokenizeChatRequest
- ]
-
- # Tokenize path delegates to OnlineRenderer.preprocess_chat,
- # where the prefix-token override lives.
- class NeMoRLServingTokenization(ServingTokenization):
- pass
-
- serving_tokenization_kwargs = dict(
- request_logger=serving_chat_kwargs["request_logger"],
- chat_template=serving_chat_kwargs["chat_template"],
- chat_template_content_format=serving_chat_kwargs[
- "chat_template_content_format"
- ],
- models=serving_chat_kwargs["models"],
- online_renderer=online_renderer,
- # ServingTokenization reads its own copy in preprocess_chat rather
- # than the renderer's, so /tokenize would otherwise render with {}
- # and diverge from /v1/chat/completions under multi-turn.
- default_chat_template_kwargs=default_chat_template_kwargs,
+ model_config=self.llm_async_engine_args.create_model_config(),
+ cfg=self.cfg,
)
- openai_serving_tokenization = NeMoRLServingTokenization(
- **serving_tokenization_kwargs
- )
-
- @app.post("/tokenize")
- async def tokenize(request: NeMoRLTokenizeRequest, raw_request: Request):
- generator = await openai_serving_tokenization.create_tokenize(
- request, raw_request
- )
-
- if isinstance(generator, ErrorResponse):
- return JSONResponse(
- content=generator.model_dump(), status_code=generator.error.code
- )
- elif isinstance(generator, TokenizeResponse):
- if (
- isinstance(request, NeMoRLTokenizeChatRequest)
- and request.ng_capture is not None
- ):
- return JSONResponse(
- content={
- "prompt_token_count": generator.count,
- "ng_context_measured": True,
- }
- )
- return JSONResponse(content=generator.model_dump())
-
- ########################################
- # Logging
- ########################################
- print(
- "Adding a vLLM logging filter so that the logs aren't spammed with not useful messages like `Added request ...`. This is to help errors pop up better and filter out noise."
- )
-
- class CleanLoggingFilter(LoggingFilter):
- def filter(self, record: LogRecord) -> bool:
- msg = record.getMessage()
-
- # vLLM does not accept `strict` tool definitions and reporting it to the user is not useful either.
- return (
- "Added request" not in msg
- and "The following fields were present in the request but ignored: {'strict'}"
- not in msg
- )
-
- vllm_async_llm_logger.addFilter(CleanLoggingFilter())
-
- from logging import getLogger as _getLogger
-
- _getLogger("vllm.entrypoints.openai.engine.protocol").addFilter(
- CleanLoggingFilter()
- )
-
- # Suppress the noisy vLLM traceback when a prompt exceeds max_model_len.
- # This is expected during multi-turn rollouts; we log a clean one-line
- # warning from _preprocess_chat instead.
- class MaxContextLengthFilter(LoggingFilter):
- def filter(self, record: LogRecord) -> bool:
- if record.exc_info and record.exc_info[1]:
- if "maximum context length" in str(record.exc_info[1]):
- return False
- return True
-
- _getLogger("vllm.entrypoints.openai.chat_completion.serving").addFilter(
- MaxContextLengthFilter()
- )
-
+ chat_server.register_routes(app, server, host=self, generation_config=self.cfg)
+ chat_server.install_log_filters()
return app
def _setup_vllm_server(self) -> "tuple[threading.Thread, str, uvicorn.Server]":
diff --git a/nemo_rl/utils/outdated_config_checks.py b/nemo_rl/utils/outdated_config_checks.py
index 5d71e3c6f42..eb90910df14 100644
--- a/nemo_rl/utils/outdated_config_checks.py
+++ b/nemo_rl/utils/outdated_config_checks.py
@@ -209,8 +209,10 @@ def reject_outdated_dynamo_router_reset_states(config: dict[str, Any]) -> None:
"metrics_include_prefixes": "set vllm_cfg.enable_vllm_metrics_logger instead",
"metrics_exclude_prefixes": "set vllm_cfg.enable_vllm_metrics_logger instead",
"worker_args": (
- "Dynamo serves only NeMo RL's own rollouts, which need no chat parsers or "
- "templates; engines are configured through vllm_cfg and vllm_kwargs"
+ "NeMo RL's chat engine in the frontend renders and parses NeMo-Gym's calls "
+ "with the native server's vllm_cfg.http_server_serving_chat_kwargs and "
+ "parser plugins, and engines are configured through vllm_cfg and "
+ "vllm_kwargs"
),
}
_REMOVED_DYNAMO_FRONTEND_KEYS = (
@@ -257,8 +259,9 @@ def reject_outdated_dynamo_backend(config: dict[str, Any]) -> None:
if key in frontend_args:
problems.append(
f"policy.generation.dynamo_cfg.frontend_args.{key} was "
- "removed: NeMo RL's rollouts send token IDs, which the "
- "frontend never tokenizes."
+ "removed: NeMo RL never uses the frontend's tokenizer: "
+ "rollouts send token IDs, and NeMo-Gym's chat calls are "
+ "rendered by NeMo RL's own chat engine."
)
if problems:
raise ValueError(" ".join(problems))
diff --git a/pyrefly.toml b/pyrefly.toml
index e0b603ad020..57de5f9c1a4 100644
--- a/pyrefly.toml
+++ b/pyrefly.toml
@@ -269,7 +269,13 @@ project-includes = [
"nemo_rl/models/generation/dynamo/config.py",
"nemo_rl/models/generation/dynamo/managed_runtime.py",
"nemo_rl/models/generation/dynamo/engine_adapter.py",
- "nemo_rl/models/generation/dynamo/vllm_adapter.py",
+ "nemo_rl/models/generation/dynamo/vllm/__init__.py",
+ "nemo_rl/models/generation/dynamo/vllm/adapter.py",
+ "nemo_rl/models/generation/dynamo/vllm/chat_engine.py",
+ "nemo_rl/models/generation/dynamo/vllm/frontend.py",
+ "nemo_rl/models/generation/dynamo/vllm/generation_cut.py",
+ "nemo_rl/models/generation/dynamo/vllm/render_pool.py",
+ "nemo_rl/models/generation/gym_call.py",
"nemo_rl/models/generation/dynamo/rollout_client.py",
"nemo_rl/models/generation/engine_supervisor.py",
"nemo_rl/models/generation/fleet_health.py",
@@ -287,6 +293,7 @@ project-includes = [
"nemo_rl/models/generation/prefix_recovery/recovery.py",
"nemo_rl/models/generation/prefix_recovery/restore.py",
"nemo_rl/models/generation/prefix_recovery/state.py",
+ "nemo_rl/models/generation/vllm/chat_server.py",
"nemo_rl/models/generation/rollout_result.py",
"nemo_rl/models/generation/capture_context.py",
"nemo_rl/models/generation/sglang/config.py",
@@ -313,6 +320,7 @@ project-includes = [
"nemo_rl/models/generation/vllm/multi_api.py",
"nemo_rl/models/generation/vllm/patches.py",
"nemo_rl/models/generation/vllm/progress_reader.py",
+ "nemo_rl/models/generation/vllm/prompt_and_capture.py",
"nemo_rl/models/generation/vllm/quantization/deepseek_v4_fp8.py",
"nemo_rl/models/generation/vllm/quantization/fp8_train_utils.py",
"nemo_rl/models/generation/vllm/quantization/mxfp8_utils.py",
diff --git a/tests/functional/L1_Functional_Tests_Dynamo.sh b/tests/functional/L1_Functional_Tests_Dynamo.sh
index 8f7e689228b..68e34b8986f 100644
--- a/tests/functional/L1_Functional_Tests_Dynamo.sh
+++ b/tests/functional/L1_Functional_Tests_Dynamo.sh
@@ -36,7 +36,18 @@ run_test() {
# Dynamo overlays: native SingleController tests rerun with policy.generation.dynamo=true.
# Each keeps the native test's checks and adds check_dynamo_overlay.sh's.
run_test fast uv run --no-sync bash ./tests/functional/grpo_dp_single_controller.sh ++policy.generation.dynamo=true
+run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller.sh ++token_capture.enabled=true ++policy.generation.dynamo=true
+# Gym's default mode: token arrays come back in the reply's nvext.nemo_rl.
+run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller.sh ++policy.generation.dynamo=true
run_test fast uv run --no-sync bash ./tests/functional/grpo_dp_single_controller_reload_refit.sh ++policy.generation.dynamo=true
+# Forest training: token-capture calls that may continue an earlier call.
+run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller_context_compaction.sh ++policy.generation.dynamo=true
+# Router replay with deferred routes: the engine stages each captured call's routed experts.
+run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller.sh ++token_capture.enabled=true policy.model_name=silence09/DeepSeek-R1-Small-2layers ++policy.router_replay.enabled=true ++token_capture.defer_routed_experts_to_policy=true ++policy.generation.dynamo=true
+# Turn recovery: Gym checkpoint + TQ snapshot, kill, restore, re-sent calls.
+run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller_turn_recovery.sh ++policy.generation.dynamo=true
+# Prefix recovery: a checkpoint cuts running calls; restore resumes them from the saved tokens.
+run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh ++policy.generation.dynamo=true
cd "${PROJECT_ROOT}/tests"
if compgen -G ".coverage*" > /dev/null; then
diff --git a/tests/functional/_gym_policy_proxy.py b/tests/functional/_gym_policy_proxy.py
index f8c1958bcc3..518580f55c2 100644
--- a/tests/functional/_gym_policy_proxy.py
+++ b/tests/functional/_gym_policy_proxy.py
@@ -46,6 +46,7 @@
from typing import Any, Optional
from nemo_rl.distributed.virtual_cluster import _get_node_ip_local
+from nemo_rl.models.generation.gym_call import GymCallExtras
from tests.functional._single_controller_hook_utils import append_event
_SKIPPED_REQUEST_HEADERS = frozenset({"host", "content-length", "connection"})
@@ -84,6 +85,12 @@ def _tool_results(body: dict[str, Any]) -> dict[str, Any]:
def _rollout_id(body: dict[str, Any]) -> Optional[str]:
capture = body.get("ng_capture")
+ if capture is None:
+ # Gym's Dynamo dialect carries it in the nemo_rl:gym: annotation.
+ extras = GymCallExtras.from_annotations(
+ (body.get("nvext") or {}).get("annotations")
+ )
+ capture = extras.ng_capture if extras is not None else None
if isinstance(capture, dict):
# Gym names it capture_key; Gyms before NVIDIA-NeMo/Gym#4128 sent rollout_id.
rollout_id = capture.get("capture_key", capture.get("rollout_id"))
diff --git a/tests/functional/_gym_prefix_recovery_phases.sh b/tests/functional/_gym_prefix_recovery_phases.sh
index 6aff50cdc49..b92d17bd19d 100644
--- a/tests/functional/_gym_prefix_recovery_phases.sh
+++ b/tests/functional/_gym_prefix_recovery_phases.sh
@@ -14,9 +14,11 @@
# Phase control shared by the prefix-recovery functional tests. Source it after
# setting PROJECT_ROOT and TEST_DIR; a test starts phase one in its own process
-# group and keeps its PID in PHASE1_PID.
+# group and keeps its PID in PHASE1_PID. A test that also runs behind Dynamo
+# keeps the log phase one writes in PHASE1_RUN_LOG.
PHASE1_PID=""
+PHASE1_RUN_LOG=""
# Hard-kill phase one and wait until its controller can no longer write
# snapshots.
@@ -37,6 +39,17 @@ stop_phase1() {
fi
sleep 1
done
+ # Behind Dynamo, the kill orphans etcd, NATS and the frontend: they run in
+ # their own sessions. Each carries phase one's DYN_NAMESPACE in its environment.
+ local namespace="" pid
+ if [[ -n "${PHASE1_RUN_LOG}" ]]; then
+ namespace=$(grep -aoE "'DYN_NAMESPACE': '[^']+'" "${PHASE1_RUN_LOG}" | head -1 | cut -d"'" -f4 || true)
+ fi
+ if [[ -n "${namespace}" ]]; then
+ for pid in $(grep -lsxz "DYN_NAMESPACE=${namespace}" /proc/[0-9]*/environ | cut -d/ -f3 || true); do
+ kill -KILL "${pid}" 2>/dev/null || true
+ done
+ fi
}
# Recover exactly the selected prefix: keep only the selected snapshot and no
diff --git a/tests/functional/check_dynamo_overlay.sh b/tests/functional/check_dynamo_overlay.sh
index 42a2b381e41..e77a178f2e2 100755
--- a/tests/functional/check_dynamo_overlay.sh
+++ b/tests/functional/check_dynamo_overlay.sh
@@ -49,6 +49,6 @@ echo "KV events the router applied: ${applied:-0}"
|| fail "The KV router applied none of the engines' KV-cache events"
dynamo_bin_dir=${NEMO_RL_DYNAMO_BIN_DIR:-/opt/dynamo/bin}
-if pgrep -f "[d]ynamo.frontend|${dynamo_bin_dir}/[e]tcd|${dynamo_bin_dir}/[n]ats-server"; then
+if pgrep -f "[d]ynamo\.vllm\.frontend|${dynamo_bin_dir}/[e]tcd|${dynamo_bin_dir}/[n]ats-server"; then
fail "Managed Dynamo services remain after shutdown"
fi
diff --git a/tests/functional/grpo_async_gym_single_controller.sh b/tests/functional/grpo_async_gym_single_controller.sh
index 15028bedb22..f10586d0910 100755
--- a/tests/functional/grpo_async_gym_single_controller.sh
+++ b/tests/functional/grpo_async_gym_single_controller.sh
@@ -147,6 +147,10 @@ if [[ "${RUN_CONVERGENCE_CHECKS:-1}" == "1" ]]; then
if [[ "$*" == *token_capture.enabled=true* ]]; then
# Nonzero only when a finalizer actor ran, i.e. capture really was on.
EXTRA_CHECKS+=('max(data["train/finalize/total_ms"]) > 0')
+ if [[ "$*" == *router_replay.enabled=true* ]]; then
+ # Every captured row carries routed experts for the policy to replay.
+ EXTRA_CHECKS+=('min(data["train/finalize/routed_experts_row_coverage"]) == 1')
+ fi
fi
if [[ "${SC_TEST_CONTEXT_COMPACTION:-0}" == "1" ]]; then
@@ -167,3 +171,7 @@ if [[ "${RUN_CONVERGENCE_CHECKS:-1}" == "1" ]]; then
'max(data["train/reward"]) > 0' \
${EXTRA_CHECKS[@]+"${EXTRA_CHECKS[@]}"}
fi
+
+if [[ "$*" == *generation.dynamo=true* ]]; then
+ bash $SCRIPT_DIR/check_dynamo_overlay.sh $RUN_LOG
+fi
diff --git a/tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh b/tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh
index f9a480a7121..bc20931b53b 100755
--- a/tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh
+++ b/tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh
@@ -204,6 +204,7 @@ setsid env \
logger.log_dir="$PHASE1_LOG_DIR" \
"$@" &
PHASE1_PID=$!
+PHASE1_RUN_LOG=$BASE_RUN_LOG
uv run --directory "$PROJECT_ROOT" --no-sync python "$SNAPSHOT_HELPER" select \
"$CHECKPOINT_DIR/bootstrap/rollout_snapshots" \
diff --git a/tests/functional/grpo_async_gym_single_controller_turn_recovery.sh b/tests/functional/grpo_async_gym_single_controller_turn_recovery.sh
index 64f2abe717c..1f22855b439 100755
--- a/tests/functional/grpo_async_gym_single_controller_turn_recovery.sh
+++ b/tests/functional/grpo_async_gym_single_controller_turn_recovery.sh
@@ -98,6 +98,16 @@ stop_phase1() {
fi
sleep 1
done
+ # Behind Dynamo, NeMo RL runs etcd, NATS and the frontend in their own
+ # sessions and stops them only on a clean exit, so the kill orphans them.
+ # Each carries phase one's DYN_NAMESPACE in its environment.
+ local namespace pid
+ namespace=$(grep -aoE "'DYN_NAMESPACE': '[^']+'" "$BASE_RUN_LOG" | head -1 | cut -d"'" -f4 || true)
+ if [[ -n "$namespace" ]]; then
+ for pid in $(grep -lsxz "DYN_NAMESPACE=$namespace" /proc/[0-9]*/environ | cut -d/ -f3 || true); do
+ kill -KILL "$pid" 2>/dev/null || true
+ done
+ fi
}
cleanup() {
diff --git a/tests/test_suites/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.sh b/tests/test_suites/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.sh
new file mode 100755
index 00000000000..e9cf7073a7c
--- /dev/null
+++ b/tests/test_suites/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.sh
@@ -0,0 +1,73 @@
+#!/bin/bash
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)
+source "${SCRIPT_DIR}/common.env"
+
+# ===== BEGIN CONFIG =====
+NUM_NODES=3
+GPUS_PER_NODE=8
+STEPS_PER_RUN=4
+MAX_STEPS=4
+NUM_RUNS=1
+NUM_MINUTES=240
+USE_GYM_CONTAINER=true
+# ===== END CONFIG =====
+
+exit_if_max_steps_reached
+
+cd "${PROJECT_ROOT}"
+uv run examples/nemo_gym/run_grpo_nemo_gym.py \
+ --config "${CONFIG_PATH}" \
+ grpo.max_num_steps="${MAX_STEPS}" \
+ logger.log_dir="${LOG_DIR}" \
+ logger.wandb_enabled=true \
+ logger.wandb.project=nemo-rl \
+ logger.wandb.name="${EXP_NAME}" \
+ logger.tensorboard_enabled=true \
+ "$@" \
+ 2>&1 | tee "${RUN_LOG}"
+
+uv run tests/json_dump_tb_logs.py "${LOG_DIR}" \
+ --output_path "${JSON_METRICS}" \
+ --require-tag-prefix generation_metrics/
+
+last_step=$(jq 'to_entries | .[] | select(.key == "train/loss") | .value | keys | map(tonumber) | max' "${JSON_METRICS}")
+if [[ ${last_step} -lt ${MAX_STEPS} ]]; then
+ echo "[ERROR] Expected step ${MAX_STEPS}, but the last train/loss step is ${last_step}"
+ exit 1
+fi
+
+uv run tests/check_metrics.py "${JSON_METRICS}" \
+ 'median(data["train/token_mult_prob_error"]) < 1.1' \
+ "data['train/token_mult_prob_error']['${MAX_STEPS}'] < 1.1" \
+ 'mean(data["train/gen_kl_error"]) < 0.02'
+
+refit_count=$(grep -c "✅ Ready for refit" "${RUN_LOG}" || true)
+cache_invalidation_count=$(grep -c \
+ "✅ Invalidated generation backend KV caches after weight update" \
+ "${RUN_LOG}" || true)
+cache_invalidation_failure_count=$(grep -c \
+ "Failed to invalidate generation backend KV caches" \
+ "${RUN_LOG}" || true)
+if [[ ${cache_invalidation_failure_count} -ne 0 ]]; then
+ echo "[ERROR] Found ${cache_invalidation_failure_count} cache invalidation failure(s)"
+ exit 1
+fi
+if [[ ${refit_count} -ne ${cache_invalidation_count} ]]; then
+ echo "[ERROR] Expected one cache invalidation per refit, but found " \
+ "${cache_invalidation_count} invalidations for ${refit_count} refits"
+ exit 1
+fi
diff --git a/tests/test_suites/nightly.txt b/tests/test_suites/nightly.txt
index 6c9552e6dd7..2b60b93ccf3 100644
--- a/tests/test_suites/nightly.txt
+++ b/tests/test_suites/nightly.txt
@@ -297,6 +297,7 @@ tests/test_suites/llm/grpo-qwen3-30ba3b-2n8g-megatron_fused_linear_logprobs.sh
tests/test_suites/llm/sft-gpt-oss-20b-1n8g-automodel-ep8.sh
# Nemotron 3 Nano 30B A3B Base BF16 tests
+tests/test_suites/llm/grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.sh
tests/test_suites/llm/sft-nanov3-30BA3B-2n8g-automodel-ep8.sh
tests/test_suites/llm/sft-nanov3-30BA3B-2n8g-automodel-lora.sh
diff --git a/tests/unit/algorithms/test_distillation.py b/tests/unit/algorithms/test_distillation.py
index f7fcabb8059..f1730f8f237 100644
--- a/tests/unit/algorithms/test_distillation.py
+++ b/tests/unit/algorithms/test_distillation.py
@@ -1346,7 +1346,7 @@ class DummyVllmGeneration:
def __init__(self, cluster, config, defer_model_load=False):
self.cfg = config
self.defer_model_load = defer_model_load
- self.dp_openai_server_base_urls = ["http://reserved-vllm"]
+ self.client_base_urls = ["http://reserved-vllm"]
# The per-job bearer token the real class generates when the
# recipe requires one; Gym's model server must receive it.
self.http_server_api_key = "k"
diff --git a/tests/unit/algorithms/test_grpo.py b/tests/unit/algorithms/test_grpo.py
index 2b3bb3abbf5..455aefbd7c7 100644
--- a/tests/unit/algorithms/test_grpo.py
+++ b/tests/unit/algorithms/test_grpo.py
@@ -4582,7 +4582,7 @@ def offload_after_refit(self):
class DummyVllmGeneration:
weight_synchronizer = None
- dp_openai_server_base_urls = ["http://vllm-worker.example/v1"]
+ client_base_urls = ["http://vllm-worker.example/v1"]
# What the real class generates when the recipe requires a key.
http_server_api_key = "per-job-key"
diff --git a/tests/unit/data/test_captured_media.py b/tests/unit/data/test_captured_media.py
index 34e77b020d5..382a4c2fd84 100644
--- a/tests/unit/data/test_captured_media.py
+++ b/tests/unit/data/test_captured_media.py
@@ -452,6 +452,7 @@ def test_splice_coordinates_handle_reasoning_shift_and_repeated_pad_runs():
model_prefix_token_ids=original,
template_prefix_token_ids=template[:6],
template_token_ids=template,
+ eos_token_id=2,
)
prompt = engine_prompt(template, [(Span(1, 2), a), (Span(7, 2), b)])
captured = capture_processed_media(
@@ -476,6 +477,7 @@ def test_continuation_captures_only_new_media_and_rejects_changed_pixels():
model_prefix_token_ids=original,
template_prefix_token_ids=template[:6],
template_token_ids=template,
+ eos_token_id=2,
)
retained = (StagedMediaTensors(**first.tensors, num_frames=None),)
prompt = engine_prompt(template, [(Span(1, 2), a), (Span(7, 2), b)])
@@ -545,6 +547,7 @@ def test_changed_retained_pad_run_cannot_steal_the_next_image():
model_prefix_token_ids=original,
template_prefix_token_ids=template_prefix,
template_token_ids=template,
+ eos_token_id=2,
)
with pytest.raises(ValueError, match="Retained media"):
capture_processed_media(
@@ -573,6 +576,7 @@ def test_span_mask_is_preserved_on_remap():
model_prefix_token_ids=[10, 77, 31, 2],
template_prefix_token_ids=[10, 31, 2],
template_token_ids=[10, 31, 2, 11, 18, 18, 12],
+ eos_token_id=2,
)
prompt = engine_prompt(
[10, 31, 2, 11, 18, 18, 12], [(Span(3, 4, mask), torch.ones(3, 2, 3))]
@@ -1173,6 +1177,7 @@ def test_video_placeholder_remap_preserves_all_timestamp_tokens():
model_prefix_token_ids=original,
template_prefix_token_ids=template_prefix,
template_token_ids=template,
+ eos_token_id=2,
)
prompt = video_prompt(template, [(Span(1, 4), frames), (Span(7, 4), frames)])
added = capture_processed_media(
@@ -1644,6 +1649,7 @@ async def test_retained_media_rejection_precedes_inference_and_survives_gym(
renderer.renderer = SimpleNamespace(
tokenizer=SimpleNamespace(decode=str, eos_token_id=2)
)
+ renderer.model_config = SimpleNamespace(max_model_len=4096)
inference = AsyncMock()
async def create_chat_completion(request, raw_request):
diff --git a/tests/unit/environments/test_nemo_gym.py b/tests/unit/environments/test_nemo_gym.py
index a2007ab4972..eab38f35c6f 100644
--- a/tests/unit/environments/test_nemo_gym.py
+++ b/tests/unit/environments/test_nemo_gym.py
@@ -2591,6 +2591,7 @@ def spin_up(handle, gym_config=None):
checkpoint_instance=GymCheckpointInstance("nemo_gym", 0),
turn_checkpointing_enabled=False,
generation_prefix_cuts_enabled=False,
+ generation_cut_control_urls=None,
)
NemoGym.__ray_metadata__.modified_class(cfg)._spinup()
return started[-1]
@@ -2651,6 +2652,7 @@ def spin_up():
checkpoint_instance=GymCheckpointInstance("nemo_gym", 0),
turn_checkpointing_enabled=False,
generation_prefix_cuts_enabled=False,
+ generation_cut_control_urls=None,
)
NemoGym.__ray_metadata__.modified_class(cfg)._spinup()
return parse_otel_resource_attributes(os.environ["OTEL_RESOURCE_ATTRIBUTES"])
diff --git a/tests/unit/environments/test_nemo_gym_token_capture.py b/tests/unit/environments/test_nemo_gym_token_capture.py
index 7c5e9d32103..2fd0049a06a 100644
--- a/tests/unit/environments/test_nemo_gym_token_capture.py
+++ b/tests/unit/environments/test_nemo_gym_token_capture.py
@@ -193,7 +193,11 @@ async def postprocess(row: dict, result: dict) -> dict:
@pytest.mark.parametrize(
("generation_backend", "expected"),
- [("vllm", "vllm_worker"), ("megatron", "megatron_worker")],
+ [
+ ("vllm", "vllm_worker"),
+ ("megatron", "megatron_worker"),
+ ("dynamo", "dynamo_worker"),
+ ],
)
def test_external_staging_backend_maps_generation_backend(
generation_backend: str, expected: str
@@ -915,30 +919,105 @@ def test_setup_nemo_gym_config_megatron_keeps_async_rollout_check_satisfied() ->
assert should_use_nemo_gym(config) is True
-def test_setup_nemo_gym_config_refuses_dynamo_serving() -> None:
- """Dynamo does not serve NeMo-Gym's calls yet, so the pairing fails at startup."""
+@pytest.mark.parametrize(
+ ("vllm_model", "expected"),
+ [
+ # NeMo RL's chat engine renders with the native default template kwargs
+ # beneath these, as the native server does; Gym sends them as written.
+ (
+ {"chat_template_kwargs": {"enable_thinking": False}},
+ {"enable_thinking": False},
+ ),
+ # The frontend answers every request carrying bad_words with a 400.
+ ({"extra_body": {"bad_words": ["x"]}}, "bad_words"),
+ # The frontend accepts stop_token_ids but drops it before our chat engine.
+ ({"extra_body": {"stop_token_ids": [7]}}, "stop_token_ids"),
+ # Gym sends sampling_overrides keys on every request too.
+ ({"sampling_overrides": {"bad_words": ["x"]}}, "sampling_overrides.bad_words"),
+ (
+ {"sampling_overrides": {"logprob_token_ids": [7]}},
+ "sampling_overrides.*logprob_token_ids",
+ ),
+ ],
+)
+def test_gym_actor_config_dynamo_turns_on_gyms_dynamo_dialect(
+ vllm_model: dict, expected: dict | str
+) -> None:
+ from responses_api_models.vllm_model.app import VLLMModelConfig
+
from nemo_rl.algorithms.grpo import MasterConfig
from nemo_rl.environments.nemo_gym import (
+ build_nemo_gym_config,
setup_nemo_gym_config,
should_use_nemo_gym,
)
+ user_vllm_model = copy.deepcopy(vllm_model)
+
config = MasterConfig.model_construct(
- env={"should_use_nemo_gym": True},
+ env={
+ "should_use_nemo_gym": True,
+ "nemo_gym": {
+ "policy_model": {"responses_api_models": {"vllm_model": vllm_model}}
+ },
+ },
data={},
policy={
"generation": {
"backend": "vllm",
"dynamo": True,
- "vllm_cfg": {"async_engine": False, "expose_http_server": False},
+ "vllm_cfg": {
+ "async_engine": True,
+ "http_server_serving_chat_kwargs": {
+ "default_chat_template_kwargs": {
+ "enable_thinking": True,
+ "truncate_history_thinking": False,
+ }
+ },
+ },
}
},
)
setup_nemo_gym_config(config, tokenizer=None)
+ assert should_use_nemo_gym(config) is True
- with pytest.raises(ValueError, match="does not serve NeMo-Gym rollouts yet"):
- should_use_nemo_gym(config)
+ def build() -> dict:
+ return build_nemo_gym_config(
+ config.env,
+ base_urls=["http://frontend"],
+ model_name="test-model",
+ enable_router_replay=False,
+ use_fastokens=False,
+ turn_checkpointing_enabled=False,
+ )["initial_global_config_dict"]
+
+ if isinstance(expected, str):
+ with pytest.raises(ValueError, match=expected):
+ build()
+ return
+ gym_config = build()
+ composed = gym_config["policy_model"]["responses_api_models"]["vllm_model"]
+ # Read back through Gym's own config, which ignores keys it does not know.
+ gym_model = VLLMModelConfig(
+ host="localhost",
+ port=8000,
+ name="policy_model",
+ entrypoint="app.py",
+ base_url="http://frontend",
+ api_key="dummy_key",
+ model="test-model",
+ return_token_id_information=True,
+ uses_reasoning_parser=True,
+ **composed,
+ )
+ assert gym_model.dynamo_frontend is True
+ # The frontend refuses reasoning under both of Gym's keys.
+ assert gym_model.reasoning_field == "reasoning_content"
+ assert gym_model.chat_template_kwargs == expected
+ assert "dynamo_dialect" not in gym_config
+ # The user's overlay is left as written.
+ assert vllm_model == user_vllm_model
def test_public_postprocess_waits_for_commit_instead_of_masking(monkeypatch) -> None:
diff --git a/tests/unit/environments/test_nemo_gym_utils.py b/tests/unit/environments/test_nemo_gym_utils.py
index 2bd5162d625..21d9b23cd7b 100644
--- a/tests/unit/environments/test_nemo_gym_utils.py
+++ b/tests/unit/environments/test_nemo_gym_utils.py
@@ -1208,6 +1208,74 @@ def test_build_nemo_gym_actors_hands_every_actor_the_policy_api_key(
assert all(cfg["policy_api_key"] == "per-job-key" for cfg in cluster.actor_configs)
+def test_build_nemo_gym_actors_gives_every_actor_the_dynamo_dialect(
+ detected_uv_dirs,
+):
+ """With Dynamo, each actor's composed config speaks Gym's Dynamo dialect.
+
+ The dialect is not a policy_model overlay, which a sharded plan would reject
+ as unowned. The frontend applies the native default template kwargs, so only
+ a shard's own kwargs reach Gym.
+ """
+ from nemo_rl.algorithms.grpo import MasterConfig
+
+ env = _shard_env_configs()
+ env["nemo_gym"]["shards"][1]["overrides"] = {
+ "policy_model": {
+ "responses_api_models": {
+ "vllm_model": {"chat_template_kwargs": {"enable_thinking": False}}
+ }
+ }
+ }
+ config = MasterConfig.model_construct(
+ env=env,
+ data={},
+ policy={
+ "generation": {
+ "backend": "vllm",
+ "dynamo": True,
+ "vllm_cfg": {
+ "http_server_serving_chat_kwargs": {
+ "default_chat_template_kwargs": {"enable_thinking": True}
+ }
+ },
+ }
+ },
+ )
+ setup_nemo_gym_config(config, tokenizer=None)
+ cluster = _FakeGymCluster()
+
+ with _patched_cluster(cluster):
+ nemo_gym_mod.build_nemo_gym_actors(
+ config.env,
+ base_urls=["http://frontend"],
+ model_name="test-model",
+ tokenizer=_TOKENIZER,
+ enable_router_replay=False,
+ use_fastokens=False,
+ turn_checkpointing_enabled=False,
+ )
+
+ template_kwargs = [
+ cfg["initial_global_config_dict"]["policy_model"]["responses_api_models"][
+ "vllm_model"
+ ].get("chat_template_kwargs")
+ for cfg in cluster.actor_configs
+ ]
+ # judged, then the tools shard's two replicas with their own override.
+ assert template_kwargs == [
+ None,
+ {"enable_thinking": False},
+ {"enable_thinking": False},
+ ]
+ for cfg in cluster.actor_configs:
+ gym_config = cfg["initial_global_config_dict"]
+ vllm_model = gym_config["policy_model"]["responses_api_models"]["vllm_model"]
+ assert vllm_model["dynamo_frontend"] is True
+ assert vllm_model["reasoning_field"] == "reasoning_content"
+ assert "dynamo_dialect" not in gym_config
+
+
def test_unsharded_checkpointing_discovers_the_actor_at_startup(detected_uv_dirs):
cluster = _FakeGymCluster()
token_capture = {
@@ -1331,8 +1399,9 @@ def as_config(config):
nemo_gym_mod._reject_typed_environment_servers(as_config(typed))
+@pytest.mark.parametrize("control_urls", [None, ["http://frontend:2400"]])
def test_prefix_checkpointing_enables_generation_cuts_on_policy_model(
- detected_uv_dirs,
+ detected_uv_dirs, control_urls
):
config = build_nemo_gym_config(
_env_configs(),
@@ -1342,6 +1411,7 @@ def test_prefix_checkpointing_enables_generation_cuts_on_policy_model(
use_fastokens=False,
turn_checkpointing_enabled=True,
generation_prefix_cuts_enabled=True,
+ generation_cut_control_urls=control_urls,
token_capture={
"enabled": True,
"capture_dir": "/capture",
@@ -1353,6 +1423,8 @@ def test_prefix_checkpointing_enables_generation_cuts_on_policy_model(
"responses_api_models"
]["vllm_model"]
assert model["checkpoint"] == {"policy": True, "generation_cuts": True}
+ # Only a backend that takes cuts elsewhere (a Dynamo frontend) names where.
+ assert model.get("generation_cut_control_url") == control_urls
@pytest.mark.nemo_gym
@@ -1369,6 +1441,7 @@ def test_prefix_checkpointing_policy_overrides_are_gym_fields(detected_uv_dirs):
use_fastokens=False,
turn_checkpointing_enabled=True,
generation_prefix_cuts_enabled=True,
+ generation_cut_control_urls=["http://frontend:2400"],
token_capture={
"enabled": True,
"capture_dir": "/capture",
@@ -1379,6 +1452,7 @@ def test_prefix_checkpointing_policy_overrides_are_gym_fields(detected_uv_dirs):
model = config["initial_global_config_dict"]["policy_model"][
"responses_api_models"
]["vllm_model"]
+ assert "generation_cut_control_url" in model
assert set(model) - set(VLLMModelConfig.model_fields) == set()
checkpoint_config = VLLMModelConfig.model_fields["checkpoint"].annotation
assert set(model["checkpoint"]) == {"policy", "generation_cuts"}
diff --git a/tests/unit/experience/test_capture_context.py b/tests/unit/experience/test_capture_context.py
index e22b9fa7932..81c70407eca 100644
--- a/tests/unit/experience/test_capture_context.py
+++ b/tests/unit/experience/test_capture_context.py
@@ -322,7 +322,8 @@ async def test_worker_managed_root_clears_untrusted_prefix_and_keeps_media_check
)
renderer = _build_server(monkeypatch, {})[0][0]
- renderer.renderer = SimpleNamespace()
+ renderer.renderer = SimpleNamespace(tokenizer=SimpleNamespace(eos_token_id=2))
+ renderer.model_config = SimpleNamespace(max_model_len=4096)
monkeypatch.setattr(
VllmAsyncGenerationWorkerImpl, "token_capture", object(), raising=False
)
diff --git a/tests/unit/models/generation/test_dynamo_chat_engine.py b/tests/unit/models/generation/test_dynamo_chat_engine.py
new file mode 100644
index 00000000000..20545d9263d
--- /dev/null
+++ b/tests/unit/models/generation/test_dynamo_chat_engine.py
@@ -0,0 +1,596 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""NeMo RL's chat engine in Dynamo's frontend, over a fake render pool and router.
+
+The render pool's own workers are covered by ``test_dynamo_render_pool.py``;
+here they are replaced by what they return, so each test checks what the
+engine does between the render and the reply.
+"""
+
+import asyncio
+import json
+from types import SimpleNamespace
+
+import pytest
+
+pytestmark = pytest.mark.vllm
+pytest.importorskip("dynamo._core")
+pytest.importorskip("vllm")
+
+from dynamo.llm.exceptions import HttpError, InvalidArgument # noqa: E402
+
+from nemo_rl.models.generation.dynamo import managed_runtime # noqa: E402
+from nemo_rl.models.generation.dynamo.vllm import chat_engine # noqa: E402
+from nemo_rl.models.generation.dynamo.vllm.render_pool import ( # noqa: E402
+ RenderedCall,
+)
+from nemo_rl.models.generation.gym_call import ( # noqa: E402
+ GYM_EXTRAS_ANNOTATION_PREFIX,
+ GymCallResult,
+)
+
+EOS = 2
+MAX_MODEL_LEN = 16
+GENERATION = {
+ "temperature": 0.7,
+ "top_p": 0.9,
+ "top_k": None,
+ "val_temperature": 0.0,
+ "val_top_p": 1.0,
+ "val_top_k": None,
+ "stop_token_ids": [EOS],
+ "vllm_cfg": {},
+}
+# Turn 2: the model's own turn 1 replaces the template's re-render of it.
+PREFIX = [1, 5, 6, EOS]
+HISTORY = [1, 7, EOS]
+TEMPLATE = [*HISTORY, 3, 4]
+SPLICED = [*PREFIX, 3, 4]
+TOKEN_IN = {"mode": "token_in", "capture_key": "rollout-1", "model_call_id": "c2"}
+
+
+def _admission(fields: dict) -> SimpleNamespace:
+ """A parsed capture admission, as the render pool returns it."""
+ return SimpleNamespace(**fields, model_dump=lambda mode="python": dict(fields))
+
+
+def _rendered(**fields) -> RenderedCall:
+ """What a render-pool worker returns for a call."""
+ return RenderedCall(
+ **{
+ "template_token_ids": [1, 3, 4],
+ "history_token_ids": None,
+ "required_prefix_token_ids": None,
+ "admission": None,
+ "serving_admission": None,
+ "verify_retained_media": False,
+ "multimodal": None,
+ "request": {"model": "model", "max_tokens": None},
+ "request_max_tokens": None,
+ "reasoning_ended": None,
+ "reasoning_parser_kwargs": None,
+ **fields,
+ }
+ )
+
+
+class _Pool:
+ def __init__(self, rendered: RenderedCall, reply: dict | None = None) -> None:
+ self.rendered = rendered
+ self.reply = reply or {
+ "choices": [
+ {
+ "message": {"role": "assistant", "content": "ok"},
+ "finish_reason": "stop",
+ }
+ ]
+ }
+ self.bodies: list[dict] = []
+ self.responded: list[tuple] = []
+ self.error: Exception | None = None
+
+ async def info(self) -> dict:
+ return {"turn_end_token_ids": [EOS], "max_model_len": MAX_MODEL_LEN}
+
+ async def render(self, body: dict, *, measure: bool = False) -> RenderedCall:
+ self.bodies.append(body)
+ if self.error is not None:
+ raise self.error
+ return self.rendered
+
+ async def respond(self, request, prompt_token_ids, output, restored=None) -> dict:
+ self.responded.append((request, prompt_token_ids, output))
+ return self.reply
+
+
+class _Item:
+ """One annotated response from Dynamo's router."""
+
+ def __init__(self, data) -> None:
+ self._data = data
+
+ def data(self):
+ return self._data
+
+
+class _Router:
+ def __init__(self, engine_data=None, error=None) -> None:
+ self.engine_data = engine_data or {
+ "output": {"text": "ok", "token_ids": [10], "finish_reason": "stop"},
+ "result": {"prompt_token_ids": [1, 3, 4], "generation_token_ids": [10]},
+ }
+ self.error = error
+ self.requests: list[dict] = []
+
+ async def generate(self, request, context=None):
+ self.requests.append(request)
+
+ async def stream():
+ if isinstance(self.error, Exception):
+ raise self.error
+ if self.error is not None:
+ # Dynamo's binding raises an engine's error frame as a ValueError.
+ raise ValueError(self.error)
+ yield _Item({"token_ids": [10], "engine_data": self.engine_data})
+
+ return stream()
+
+
+class _Capture:
+ """The TQ prefix source: returns the staged prefix for a token_in admission."""
+
+ def __init__(self) -> None:
+ self.resolved: list[dict] = []
+
+ def resolve_prefix(self, serving_admission) -> list[int]:
+ self.resolved.append(serving_admission.model_dump(mode="json"))
+ return PREFIX
+
+
+def _call(extras: dict | None = None, **fields) -> dict:
+ """A chat call as Dynamo's frontend hands it over: unset fields are None."""
+ nvext = None
+ if extras is not None:
+ nvext = {"annotations": [GYM_EXTRAS_ANNOTATION_PREFIX + json.dumps(extras)]}
+ return {
+ "model": "model",
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": "look"},
+ {"type": "image_url", "image_url": {"url": "u", "detail": None}},
+ ],
+ }
+ ],
+ "temperature": 0.7,
+ "top_p": 0.9,
+ "top_k": None,
+ "n": None,
+ "min_p": None,
+ "presence_penalty": None,
+ "logit_bias": None,
+ "max_tokens": 20,
+ "chat_template_args": None,
+ "unsupported_fields": None,
+ "nvext": nvext,
+ **fields,
+ }
+
+
+def _serve(pool, router, request, capture=None) -> list[dict]:
+ engine = chat_engine.NemoRLChatEngine(
+ pool=pool,
+ routed_engine=router,
+ spec={"generation": GENERATION},
+ capture=capture or _Capture(),
+ )
+
+ async def run():
+ return [chunk async for chunk in engine.generate(request, None)]
+
+ return asyncio.run(run())
+
+
+def test_a_gym_call_is_rendered_spliced_and_routed_with_its_capture_plan() -> None:
+ pool = _Pool(
+ _rendered(
+ template_token_ids=TEMPLATE,
+ history_token_ids=HISTORY,
+ required_prefix_token_ids=PREFIX,
+ request={"model": "model", "max_tokens": 20, "max_completion_tokens": None},
+ request_max_tokens=20,
+ reasoning_ended=False,
+ ),
+ reply={
+ "choices": [
+ {
+ "message": {
+ "role": "assistant",
+ "content": None,
+ "reasoning": "thinking",
+ "tool_calls": [
+ {
+ "id": "call-0",
+ "type": "function",
+ "function": {"name": "f", "arguments": "{}"},
+ }
+ ],
+ },
+ "finish_reason": "tool_calls",
+ }
+ ],
+ "usage": {"prompt_tokens": 6, "completion_tokens": 1, "total_tokens": 7},
+ },
+ )
+ router = _Router()
+
+ (chunk,) = _serve(
+ pool,
+ router,
+ _call(
+ {"required_prefix_token_ids": PREFIX},
+ chat_template_args={"enable_thinking": True},
+ ),
+ )
+
+ # The worker renders the call without Dynamo's None fields, with the
+ # profile's exact sampling and Gym's extras as the native request fields.
+ (body,) = pool.bodies
+ assert body["messages"][0]["content"][1] == {
+ "type": "image_url",
+ "image_url": {"url": "u"},
+ }
+ assert {"n", "logit_bias", "nvext", "chat_template_args"}.isdisjoint(body)
+ assert body["chat_template_kwargs"] == {"enable_thinking": True}
+ assert (body["temperature"], body["top_p"], body["top_k"]) == (0.7, 0.9, -1)
+ assert (body["min_p"], body["repetition_penalty"]) == (0.0, 1.0)
+ assert (body["ng_capture"], body["required_prefix_token_ids"]) == (None, PREFIX)
+ # The router gets the spliced prompt, the budget clamped to the context,
+ # and the plan the engine's adapter serves the call by.
+ (routed,) = router.requests
+ assert routed["token_ids"] == SPLICED
+ assert routed["stop_conditions"] == {"max_tokens": MAX_MODEL_LEN - len(SPLICED)}
+ assert routed["annotations"] == [chat_engine.GYM_CALL_ANNOTATION]
+ wire = routed["extra_args"][chat_engine.CALL_KEY]
+ plan = wire["prepared_prompt"]
+ # The request as rendered; the engine sets the plan's clamped budget.
+ assert wire["request"] == {
+ "model": "model",
+ "max_tokens": 20,
+ "max_completion_tokens": None,
+ }
+ assert plan["max_tokens"] == MAX_MODEL_LEN - len(SPLICED)
+ assert (plan["splice"], plan["prefix_len"]) == (
+ # Each side is cut at its turn end: PREFIX[:3] + TEMPLATE[2:].
+ [3, 2],
+ len(PREFIX),
+ )
+ # A text call sends its prompt once: the template's ids ride only with media.
+ assert "template_token_ids" not in plan
+ assert (plan["admission"], wire["multimodal"], wire["reasoning_ended"]) == (
+ None,
+ None,
+ False,
+ )
+ # The reply is vLLM's, built from the engine's output, with the result in nvext.
+ ((_, prompt_token_ids, output),) = pool.responded
+ assert (prompt_token_ids, output) == (SPLICED, router.engine_data["output"])
+ (choice,) = chunk["choices"]
+ assert choice["delta"]["reasoning_content"] == "thinking"
+ assert choice["delta"]["tool_calls"][0]["index"] == 0
+ assert choice["finish_reason"] == "tool_calls"
+ assert chunk["nvext"] == {"nemo_rl": router.engine_data["result"]}
+ assert chunk["usage"]["prompt_tokens"] == 6
+
+
+def test_a_token_in_call_splices_the_prefix_staged_in_tq() -> None:
+ pool = _Pool(
+ _rendered(
+ template_token_ids=TEMPLATE,
+ history_token_ids=HISTORY,
+ admission=_admission(TOKEN_IN),
+ serving_admission=_admission(TOKEN_IN),
+ )
+ )
+ router = _Router()
+ capture = _Capture()
+
+ _serve(pool, router, _call({"ng_capture": TOKEN_IN}), capture)
+
+ assert pool.bodies[0]["ng_capture"] == TOKEN_IN
+ assert capture.resolved == [TOKEN_IN]
+ (routed,) = router.requests
+ assert routed["token_ids"] == SPLICED
+ plan = routed["extra_args"][chat_engine.CALL_KEY]["prepared_prompt"]
+ assert (plan["admission"], plan["serving_admission"], plan["prefix_len"]) == (
+ TOKEN_IN,
+ TOKEN_IN,
+ len(PREFIX),
+ )
+
+
+def test_a_measure_call_returns_the_prompt_length_without_generating() -> None:
+ pool = _Pool(
+ _rendered(
+ template_token_ids=TEMPLATE,
+ history_token_ids=HISTORY,
+ required_prefix_token_ids=PREFIX,
+ )
+ )
+ router = _Router()
+
+ (chunk,) = _serve(
+ pool,
+ router,
+ _call({"required_prefix_token_ids": PREFIX, "ng_measure": True}),
+ )
+
+ # The length of the prompt the call would run, as spliced.
+ assert chunk["nvext"] == {
+ "nemo_rl": GymCallResult(
+ prompt_token_count=len(SPLICED), ng_context_measured=True
+ ).model_dump(exclude_none=True)
+ }
+ assert (router.requests, pool.responded) == ([], [])
+
+
+@pytest.mark.parametrize(
+ ("request_fields", "rendered", "error"),
+ [
+ ({"extras": None}, {}, "nemo_rl:gym: annotation"),
+ ({"temperature": 0.5}, {}, "matches neither"),
+ ({"presence_penalty": 0.5}, {}, "presence_penalty=0.5"),
+ ({"n": 2}, {}, "n=1"),
+ ({"stream": True}, {}, "stream=true"),
+ # The echoed prefix, spliced in, leaves no room for output.
+ (
+ {},
+ {
+ "template_token_ids": TEMPLATE,
+ "history_token_ids": HISTORY,
+ "required_prefix_token_ids": [1, *range(20, 20 + MAX_MODEL_LEN), EOS],
+ },
+ "maximum context length",
+ ),
+ ],
+)
+def test_gym_calls_nemo_rl_cannot_serve_are_refused_with_a_400(
+ request_fields, rendered, error
+) -> None:
+ pool = _Pool(_rendered(**rendered))
+ router = _Router()
+ request_fields = {"extras": {}, **request_fields}
+
+ with pytest.raises(HttpError, match=error) as refused:
+ _serve(pool, router, _call(**request_fields))
+
+ assert refused.value.code == 400
+ assert router.requests == []
+
+
+@pytest.mark.nemo_gym
+@pytest.mark.parametrize("captured", [True, False])
+@pytest.mark.parametrize("refused_by", ["render", "clamp", "engine"])
+def test_a_call_refused_before_it_runs_acknowledges_its_capture_to_gym(
+ captured, refused_by
+) -> None:
+ from nemo_gym.token_id_capture.external_capture import (
+ split_dynamo_capture_refusal,
+ )
+ from vllm.exceptions import VLLMValidationError
+
+ pool = _Pool(_rendered())
+ router = _Router()
+ if refused_by == "render":
+ pool.error = VLLMValidationError("maximum context length is 16 tokens")
+ elif refused_by == "clamp":
+ # The prompt alone fills the context.
+ pool.rendered = _rendered(template_token_ids=list(range(MAX_MODEL_LEN)))
+ else:
+ router.engine_data = {
+ "refusal": {
+ "status": 400,
+ "message": "maximum context length is 16 tokens",
+ "acknowledge": True,
+ }
+ }
+ extras = {"ng_capture": TOKEN_IN} if captured else {}
+
+ with pytest.raises(HttpError) as refused:
+ _serve(pool, router, _call(extras))
+
+ # Gym reads the acknowledgement off the message, as natively off the header.
+ message, refused_call = split_dynamo_capture_refusal(refused.value.message)
+ assert (refused.value.code, "maximum context length" in message) == (400, True)
+ assert refused_call == (TOKEN_IN["model_call_id"] if captured else None)
+
+
+@pytest.mark.parametrize(
+ ("router", "status", "message"),
+ [
+ (
+ _Router(error="BackendInvalidArgument: RuntimeError: engine died"),
+ 500,
+ "engine died",
+ ),
+ # From Dynamo 1.7, the engine adapter's InvalidArgument (a RuntimeError it
+ # wrapped, e.g. non-finite log-probs) is raised as itself, not a ValueError.
+ (
+ _Router(error=InvalidArgument("RuntimeError: non-finite log-prob (nan)")),
+ 500,
+ "non-finite log-prob",
+ ),
+ # Refused before it ran, with the status the native server answers.
+ (
+ _Router(
+ engine_data={
+ "refusal": {
+ "status": 503,
+ "message": "queue full",
+ "acknowledge": False,
+ }
+ }
+ ),
+ 503,
+ "queue full",
+ ),
+ ],
+ ids=["error", "engine-invalid-argument", "refusal"],
+)
+def test_an_engine_that_does_not_serve_the_call_fails_it(router, status, message):
+ with pytest.raises(HttpError, match=message) as failed:
+ _serve(_Pool(_rendered()), router, _call({}))
+
+ assert failed.value.code == status
+
+
+def test_dynamos_frontend_still_builds_its_entrypoint_by_the_name_nemo_rl_wraps():
+ import dynamo.frontend.main as frontend_main
+
+ # frontend.main() swaps this module global for one that adds NeMo RL's
+ # chat_engine_factory; if Dynamo stops reading it there, Dynamo's own
+ # preprocessor serves chat and every NeMo-Gym call is refused.
+ assert "EntrypointArgs" in frontend_main.async_main.__code__.co_names
+ # It swaps this one to serve NeMo-Gym's generation cuts from the frontend's
+ # runtime; if Dynamo stops reading it there, every cut gets a 404.
+ assert "DistributedRuntime" in frontend_main.async_main.__code__.co_names
+
+
+def test_only_nemo_rls_engines_are_served() -> None:
+ model = SimpleNamespace(
+ runtime_config=lambda: {"runtime_data": {}},
+ name=lambda: "other-model",
+ local_dir=lambda: "/snapshots/other",
+ )
+
+ async def build():
+ return await chat_engine.ChatEngineFactory().chat_engine_factory(
+ 0, model, _Router()
+ )
+
+ with pytest.raises(RuntimeError, match="without NeMo RL's chat settings"):
+ asyncio.run(build())
+
+
+@pytest.mark.parametrize("exposes_chat", [True, False])
+def test_render_workers_start_at_registration_only_where_chat_is_served(
+ monkeypatch, exposes_chat: bool
+) -> None:
+ started: list[int] = []
+
+ class _StartedPool:
+ def __init__(self, *, workers: int) -> None:
+ self._workers = workers
+
+ def build(self, spec: dict, model_dir: str) -> None:
+ pass
+
+ async def start(self) -> None:
+ started.append(self._workers)
+
+ monkeypatch.setattr(chat_engine, "RenderPool", _StartedPool)
+ monkeypatch.setenv(chat_engine.RENDER_WORKERS_ENV, "3")
+ # NeMo-Gym needs the flag, and the native worker starts its server on it.
+ generation = {**GENERATION, "vllm_cfg": {"expose_http_server": exposes_chat}}
+ spec = {"engine_args": {}, "generation": generation}
+ model = SimpleNamespace(
+ runtime_config=lambda: {"runtime_data": {chat_engine.RUNTIME_DATA_KEY: spec}},
+ name=lambda: "model",
+ local_dir=lambda: "/snapshots/model",
+ )
+
+ async def register() -> None:
+ await chat_engine.ChatEngineFactory().chat_engine_factory(0, model, _Router())
+ # The start runs in the background; registration never waits for it.
+ await asyncio.gather(*(asyncio.all_tasks() - {asyncio.current_task()}))
+
+ asyncio.run(register())
+
+ assert started == ([3] if exposes_chat else [])
+
+
+@pytest.mark.parametrize(
+ "sent_early, kept, built_from",
+ [
+ # The engine registers the policy.generation it sent before it loaded.
+ ("the same generation", True, "/snapshots/early"),
+ ("another generation", False, "/snapshots/registered"),
+ # Nothing came before registration: the waiting workers build then.
+ (None, True, "/snapshots/registered"),
+ ],
+)
+def test_registration_keeps_render_workers_built_from_the_same_settings(
+ monkeypatch, tmp_path, sent_early, kept: bool, built_from: str
+) -> None:
+ pools: list = []
+
+ class _Pool:
+ def __init__(self, *, workers: int) -> None:
+ self.built: list[tuple[dict, str]] = []
+ self.shut_down = False
+ pools.append(self)
+
+ def build(self, spec: dict, model_dir: str) -> None:
+ self.built.append((spec, model_dir))
+
+ async def start(self) -> None:
+ pass
+
+ def shutdown(self) -> None:
+ self.shut_down = True
+
+ async def early_model_dir(model: str) -> str:
+ return "/snapshots/early"
+
+ monkeypatch.setattr(chat_engine, "RenderPool", _Pool)
+ monkeypatch.setattr(chat_engine, "weight_free_model_path", early_model_dir)
+ monkeypatch.setenv(chat_engine.RENDER_WORKERS_ENV, "3")
+ settings_file = tmp_path / "chat_settings.json"
+ monkeypatch.setenv(managed_runtime.CHAT_SETTINGS_FILE_ENV, str(settings_file))
+ generation = {
+ **GENERATION,
+ "model_name": "model",
+ "vllm_cfg": {"expose_http_server": True},
+ }
+ spec = {"engine_args": {}, "generation": generation}
+ if sent_early is not None:
+ early = {
+ # vLLM fills in more engine args while it builds the engine.
+ "engine_args": {"max_num_seqs": None},
+ "generation": {
+ **generation,
+ "top_p": 0.9 if sent_early == "the same generation" else 0.5,
+ },
+ }
+ settings_file.write_text(json.dumps({chat_engine.RUNTIME_DATA_KEY: early}))
+ model = SimpleNamespace(
+ runtime_config=lambda: {"runtime_data": {chat_engine.RUNTIME_DATA_KEY: spec}},
+ name=lambda: "model",
+ local_dir=lambda: "/snapshots/registered",
+ )
+
+ async def launch_then_register() -> None:
+ factory = chat_engine.ChatEngineFactory()
+ factory.start_render_workers()
+ while sent_early is not None and not pools[0].built:
+ await asyncio.sleep(0.01)
+ await factory.chat_engine_factory(0, model, _Router())
+
+ asyncio.run(launch_then_register())
+
+ assert pools[0].shut_down is not kept
+ assert len(pools) == (1 if kept else 2)
+ ((built_spec, model_dir),) = pools[-1].built
+ assert (built_spec["generation"], model_dir) == (generation, built_from)
diff --git a/tests/unit/models/generation/test_dynamo_config.py b/tests/unit/models/generation/test_dynamo_config.py
index baae91f105e..1ad2e98a007 100644
--- a/tests/unit/models/generation/test_dynamo_config.py
+++ b/tests/unit/models/generation/test_dynamo_config.py
@@ -51,7 +51,8 @@ def test_dynamo_true_alone_is_a_complete_config(dynamo: bool) -> None:
config = configure_generation_config(_vllm_config(dynamo=dynamo), _Tokenizer())
# Dynamo serves the AsyncLLM, so the switch selects the async engine.
- # Rollouts read token IDs only, so the engine skips its tokenizer as natively.
+ # Rollouts read token IDs only, so the engine skips its tokenizer as natively;
+ # NeMo-Gym runs load it through expose_http_server, which Gym requires.
assert config["vllm_cfg"]["async_engine"] is dynamo
assert config["vllm_cfg"]["skip_tokenizer_init"] is True
assert "dynamo_cfg" not in config
@@ -96,6 +97,21 @@ def test_dynamo_true_alone_is_a_complete_config(dynamo: bool) -> None:
),
"checks no bearer token",
),
+ # Gym's calls go through the Dynamo frontend, not vLLM's HTTP frontends.
+ (
+ _vllm_config(
+ dynamo=True,
+ vllm_cfg={
+ "async_engine": True,
+ "expose_http_server": True,
+ "max_model_len": 512,
+ "tensor_parallel_size": 1,
+ "expert_parallel_size": 1,
+ "http_api_server_count": 2,
+ },
+ ),
+ "would serve none of them",
+ ),
# Direct routing needs a worker ID that rollouts never carry.
(
_vllm_config(
@@ -103,6 +119,16 @@ def test_dynamo_true_alone_is_a_complete_config(dynamo: bool) -> None:
),
"router_mode",
),
+ # NeMo-Gym's chat calls are NeMo RL's chat engine's, never Dynamo's.
+ (
+ _vllm_config(
+ dynamo=True,
+ dynamo_cfg={
+ "frontend_args": {"extra_cli_args": ["--dyn-chat-processor=vllm"]}
+ },
+ ),
+ "may not set --dyn-chat-processor",
+ ),
],
)
def test_configs_dynamo_would_not_serve_as_written_fail_at_startup(
@@ -120,7 +146,15 @@ def test_frontend_argv_is_runtime_owned() -> None:
)
argv = frontend_argv(host="0.0.0.0", port=3001, namespace="nemo-rl", dynamo_cfg=cfg)
assert argv[argv.index("--router-mode") + 1] == "kv"
+ # Gym names a token-capture call's session; keep the session on one engine.
+ assert argv[argv.index("--router-session-affinity-ttl-secs") + 1] == "300"
assert argv[-2:] == ["--foo", "1"]
+ no_affinity = DynamoCfg.model_validate(
+ {"frontend_args": {"session_affinity_ttl_secs": None}}
+ )
+ assert "--router-session-affinity-ttl-secs" not in frontend_argv(
+ host="0.0.0.0", port=3001, namespace="nemo-rl", dynamo_cfg=no_affinity
+ )
clashing = DynamoCfg(
frontend_args=DynamoFrontendArgs(extra_cli_args=["--http-port", "9"])
diff --git a/tests/unit/models/generation/test_dynamo_generation_cut.py b/tests/unit/models/generation/test_dynamo_generation_cut.py
new file mode 100644
index 00000000000..7c960fc6f3f
--- /dev/null
+++ b/tests/unit/models/generation/test_dynamo_generation_cut.py
@@ -0,0 +1,192 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""NeMo-Gym's checkpoint generation cut through the Dynamo frontend."""
+
+import asyncio
+import time
+
+import pytest
+
+pytest.importorskip("nemo_gym")
+
+from nemo_gym._checkpoint.generation_cut import ( # noqa: E402
+ GenerationCutInventory,
+ GenerationCutPrefix,
+ GenerationCutPrefixAck,
+)
+
+from nemo_rl.models.generation.dynamo import managed_runtime # noqa: E402
+from nemo_rl.models.generation.dynamo.config import DynamoCfg # noqa: E402
+from nemo_rl.models.generation.dynamo.managed_runtime import ( # noqa: E402
+ ManagedDynamoRuntime,
+)
+from nemo_rl.models.generation.dynamo.vllm.generation_cut import ( # noqa: E402
+ CUT_PREFIXES_KEY,
+ DEADLINE_KEY,
+ GENERATION_CUT_ANNOTATION,
+ INVENTORY_KEY,
+ EngineCutFanOut,
+)
+
+pytestmark = pytest.mark.nemo_gym
+
+
+class _Item:
+ def __init__(self, data) -> None:
+ self._data = data
+
+ def data(self):
+ return self._data
+
+
+class _Late(list):
+ """An engine's answer that reaches the frontend after its deadline."""
+
+
+class _Client:
+ """Dynamo's request-plane client: each engine answers a direct request."""
+
+ def __init__(self, engines: dict[int, object]) -> None:
+ self._engines = engines
+ self.requests: dict[int, dict] = {}
+
+ def instance_ids(self) -> list[int]:
+ return list(self._engines)
+
+ async def direct(self, request, instance_id):
+ self.requests[instance_id] = request
+ answer = self._engines[instance_id]
+ if isinstance(answer, Exception):
+ raise answer
+
+ async def stream():
+ if isinstance(answer, _Late):
+ await asyncio.sleep(
+ request["extra_args"][DEADLINE_KEY] - time.time() + 5
+ )
+ yield _Item({"token_ids": [], "engine_data": {CUT_PREFIXES_KEY: answer}})
+
+ return stream()
+
+
+class _Runtime:
+ def __init__(self, client: _Client) -> None:
+ self._client = client
+ self.paths: list[str] = []
+
+ def endpoint(self, path: str) -> "_Runtime":
+ self.paths.append(path)
+ return self
+
+ def client(self) -> "asyncio.Future[_Client]":
+ # A future, as Dynamo's binding returns: not a coroutine.
+ future = asyncio.get_running_loop().create_future()
+ future.set_result(self._client)
+ return future
+
+
+def _call(name: str) -> GenerationCutPrefix:
+ return GenerationCutPrefix(
+ ticket_id=f"ticket-{name}",
+ rollout_id="r0",
+ attempt=0,
+ model_call_id=name,
+ admitted_at=1.0,
+ )
+
+
+def test_a_cut_reaches_every_engine_and_each_call_gets_its_engines_answer(
+ monkeypatch, tmp_path
+) -> None:
+ held, lost, late = _call("held"), _call("lost"), _call("late")
+ inventory = GenerationCutInventory.build(
+ checkpoint_id="checkpoint-1",
+ server_name="policy_model",
+ active_prefixes=[held, lost, late],
+ )
+
+ def cut(call: GenerationCutPrefix) -> dict:
+ return GenerationCutPrefixAck(
+ **call.model_dump(),
+ disposition="durable_prefix",
+ cut_kind="active_prefix",
+ frozen_buffer_id="active/checkpoint-1/0",
+ staging_keys=(
+ f"__generation_cut__/checkpoint-1/r0/{call.model_call_id}/0",
+ ),
+ prefix_token_count=3,
+ prefix_digest="0" * 64,
+ effective_output_limit=8,
+ ).model_dump(mode="json")
+
+ client = _Client(
+ {
+ 1: [], # holds none of the calls
+ 2: [cut(held)],
+ 3: RuntimeError("engine gone"),
+ 4: _Late([cut(late)]),
+ }
+ )
+ runtime = _Runtime(client)
+ monkeypatch.setenv("DYN_NAMESPACE", "nemo-rl-test")
+ monkeypatch.setenv(managed_runtime.CAPTURE_FILE_ENV, str(tmp_path / "capture.pkl"))
+ driver = ManagedDynamoRuntime(
+ dynamo_cfg=DynamoCfg(),
+ model_name="model",
+ frontend_python="python",
+ ray_address="10.0.0.1:6379",
+ serves_chat=True,
+ )
+ driver._chat_dir = str(tmp_path)
+
+ async def run():
+ fan_out = EngineCutFanOut(runtime, asyncio.get_running_loop())
+ # Before capture is set up the route refuses every cut, as natively.
+ assert fan_out.control_token is None
+ driver.publish_chat_capture(
+ dp_cfg={"backend": "simple"},
+ staging_partition="staging",
+ capture_media=False,
+ generation_cut_control_token="secret",
+ generation_cut_control_timeout_s=0.5,
+ )
+ assert fan_out.control_token == "secret"
+ arrived = time.time()
+ return arrived, await fan_out.run_cut(inventory)
+
+ arrived, receipt = asyncio.run(run())
+
+ # Gym accepts it as the answer to its inventory: every call acked once.
+ receipt.validate_for(inventory)
+ assert {ack.model_call_id: ack.disposition for ack in receipt.prefixes} == {
+ "held": "durable_prefix",
+ "lost": "durable_failure",
+ # Its engine cut it, but answered after the frontend stopped waiting.
+ "late": "durable_failure",
+ }
+ assert runtime.paths == ["nemo-rl-test.backend.generate"]
+ # Every engine gets the whole inventory and the frontend's one deadline,
+ # counted from the cut's arrival; token_ids is always present.
+ deadline = client.requests[1]["extra_args"][DEADLINE_KEY]
+ assert arrived + 0.5 <= deadline <= time.time()
+ request = {
+ "token_ids": [],
+ "annotations": [GENERATION_CUT_ANNOTATION],
+ "extra_args": {
+ INVENTORY_KEY: inventory.model_dump(mode="json"),
+ DEADLINE_KEY: deadline,
+ },
+ }
+ assert client.requests == {1: request, 2: request, 3: request, 4: request}
diff --git a/tests/unit/models/generation/test_dynamo_managed_runtime.py b/tests/unit/models/generation/test_dynamo_managed_runtime.py
index c6dde7fcf51..446c6cc77f8 100644
--- a/tests/unit/models/generation/test_dynamo_managed_runtime.py
+++ b/tests/unit/models/generation/test_dynamo_managed_runtime.py
@@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import json
+import pickle
import signal
import subprocess
import sys
@@ -19,6 +20,7 @@
import pytest
+from nemo_rl.models.generation.dynamo import managed_runtime
from nemo_rl.models.generation.dynamo.config import DynamoCfg, DynamoFrontendArgs
from nemo_rl.models.generation.dynamo.managed_runtime import (
ManagedDynamoRuntime,
@@ -27,13 +29,15 @@
)
-def _runtime(**frontend_args) -> ManagedDynamoRuntime:
+def _runtime(serves_chat: bool = False, **frontend_args) -> ManagedDynamoRuntime:
return ManagedDynamoRuntime(
dynamo_cfg=DynamoCfg(
startup_timeout_s=5, frontend_args=DynamoFrontendArgs(**frontend_args)
),
model_name="model",
frontend_python="/venvs/vllm/bin/python",
+ ray_address="10.0.0.1:6379",
+ serves_chat=serves_chat,
)
@@ -285,19 +289,65 @@ def check_engines() -> None:
runtime.wait_for_engines(1, check_engines=check_engines)
-def test_frontend_launches_from_the_given_interpreter(monkeypatch) -> None:
- runtime = _runtime()
+@pytest.mark.parametrize("serves_chat", [True, False])
+def test_frontend_launches_nemo_rls_chat_engine_from_the_given_interpreter(
+ monkeypatch, serves_chat: bool
+) -> None:
+ runtime = _runtime(serves_chat, render_workers=3)
runtime._frontend_port = 3000
+ runtime._generation_cut_port = 2400
runtime._namespace = "nemo-rl-test"
launched = []
monkeypatch.setattr(
"nemo_rl.models.generation.dynamo.managed_runtime.subprocess.Popen",
- lambda command, **kwargs: launched.append(command) or _FakeProcess(),
+ lambda command, **kwargs: launched.append((command, kwargs["env"]))
+ or _FakeProcess(),
)
runtime._start_frontend()
+ runtime.publish_chat_capture(
+ dp_cfg={"backend": "simple"},
+ staging_partition="staging",
+ capture_media=False,
+ generation_cut_control_token="secret",
+ generation_cut_control_timeout_s=60.0,
+ )
- assert launched[0][:3] == ["/venvs/vllm/bin/python", "-m", "dynamo.frontend"]
+ ((command, env),) = launched
+ # Dynamo's frontend, with NeMo RL's chat engine serving chat calls.
+ assert command[:3] == [
+ "/venvs/vllm/bin/python",
+ "-m",
+ "nemo_rl.models.generation.dynamo.vllm.frontend",
+ ]
+ assert env[managed_runtime.RENDER_WORKERS_ENV] == "3"
+ assert env[managed_runtime.RAY_ADDRESS_ENV] == "10.0.0.1:6379"
+ # NeMo-Gym's checkpoint generation cuts go to a port of the frontend's own.
+ assert env[managed_runtime.GENERATION_CUT_PORT_ENV] == "2400"
+ # Capture is set up after the engines register; the chat engine reads it
+ # from the file the frontend was told about.
+ with open(env[managed_runtime.CAPTURE_FILE_ENV], "rb") as capture_file:
+ assert pickle.load(capture_file) == {
+ "data_plane": {"backend": "simple"},
+ "staging_partition": "staging",
+ "capture_media": False,
+ "generation_cut_control_token": "secret",
+ "generation_cut_control_timeout_s": 60.0,
+ }
+ # Only a run that serves chat names the file an engine's chat settings
+ # arrive in before it loads; the frontend starts its render workers on it.
+ assert (managed_runtime.CHAT_SETTINGS_FILE_ENV in env) is serves_chat
+ if serves_chat:
+ monkeypatch.setenv(
+ managed_runtime.CHAT_SETTINGS_FILE_ENV,
+ env[managed_runtime.CHAT_SETTINGS_FILE_ENV],
+ )
+ assert managed_runtime.read_chat_settings() is None
+ runtime.publish_chat_settings({"nemo_rl_chat": {"engine_args": {}}})
+ assert managed_runtime.read_chat_settings() == {
+ "nemo_rl_chat": {"engine_args": {}}
+ }
+ runtime.shutdown()
def test_shutdown_logs_the_kv_events_the_router_applied(monkeypatch, capsys) -> None:
diff --git a/tests/unit/models/generation/test_dynamo_render_pool.py b/tests/unit/models/generation/test_dynamo_render_pool.py
new file mode 100644
index 00000000000..5a2a4513232
--- /dev/null
+++ b/tests/unit/models/generation/test_dynamo_render_pool.py
@@ -0,0 +1,388 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""The Dynamo frontend's render pool, over a tiny Qwen3 checkpoint.
+
+The pool is spawned as in the frontend, so the spawn context, pickling and
+parser registration are exercised; renders are checked against the
+checkpoint's own chat template, replies against vLLM's Qwen3 parsers.
+"""
+
+import asyncio
+import json
+import os
+import signal
+import time
+from concurrent.futures import ThreadPoolExecutor
+from concurrent.futures.process import BrokenProcessPool
+
+import pytest
+
+pytestmark = pytest.mark.vllm
+pytest.importorskip("vllm")
+
+from transformers import AutoTokenizer # noqa: E402
+
+from nemo_rl.models.generation.dynamo.vllm.render_pool import ( # noqa: E402
+ RenderPool,
+ _wait_for_settings,
+ _Worker,
+)
+from nemo_rl.models.generation.prefix_recovery.restore import ( # noqa: E402
+ _classify_restored_prefix_terminal,
+)
+from tests.unit.models.generation.prefix_restore_cases import ( # noqa: E402
+ RESTORE_CASES,
+ RestoreCase,
+)
+from tests.unit.models.generation.test_prefix_restore_cases import ( # noqa: E402
+ _engine_text,
+ _ids,
+ _tokenizer,
+)
+
+MAX_MODEL_LEN = 512
+SPEC = {
+ # The engine's own args, as it publishes them at registration.
+ "engine_args": {"max_model_len": MAX_MODEL_LEN},
+ "generation": {
+ "temperature": 0.7,
+ "top_p": 0.9,
+ "top_k": None,
+ "val_temperature": 0.0,
+ "val_top_p": 1.0,
+ "val_top_k": None,
+ "stop_token_ids": None,
+ "vllm_cfg": {
+ "http_server_serving_chat_kwargs": {
+ "tool_parser": "hermes",
+ "reasoning_parser": "qwen3",
+ "enable_auto_tools": True,
+ }
+ },
+ },
+}
+TOOLS = [
+ {
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "The weather in a city.",
+ "parameters": {
+ "type": "object",
+ "properties": {"city": {"type": "string"}},
+ "required": ["city"],
+ },
+ },
+ }
+]
+MESSAGES = [
+ {"role": "user", "content": "Weather in Paris?"},
+ {
+ "role": "assistant",
+ "content": "",
+ "tool_calls": [
+ {
+ "id": "call-0",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'},
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "call-0", "content": "sunny"},
+]
+
+
+def _template_ids(tokenizer, messages, *, add_generation_prompt: bool) -> list[int]:
+ text = tokenizer.apply_chat_template(
+ messages,
+ tools=TOOLS,
+ add_generation_prompt=add_generation_prompt,
+ tokenize=False,
+ )
+ return tokenizer.encode(text, add_special_tokens=False)
+
+
+def _pool(spec: dict, model_dir: str) -> RenderPool:
+ pool = RenderPool(workers=1)
+ pool.build(spec, model_dir)
+ return pool
+
+
+def test_the_pool_renders_and_answers_with_the_checkpoints_template_and_parsers(
+ tiny_qwen3_model_path,
+) -> None:
+ tokenizer = AutoTokenizer.from_pretrained(tiny_qwen3_model_path)
+ reply_text = (
+ "\nchecking\n\n\n\n"
+ '{"name": "get_weather", "arguments": {"city": "Lyon"}}\n'
+ )
+ reply_ids = tokenizer.encode(reply_text, add_special_tokens=False)
+
+ async def serve():
+ pool = _pool(SPEC, tiny_qwen3_model_path)
+ try:
+ info = await pool.info()
+ rendered = await pool.render(
+ {
+ "model": "model",
+ "messages": MESSAGES,
+ "tools": TOOLS,
+ "max_tokens": 64,
+ "temperature": 0.7,
+ "top_p": 0.9,
+ # Gym's echo of the last assistant turn, without capture.
+ "required_prefix_token_ids": [1, 2, 3],
+ }
+ )
+ reply = await pool.respond(
+ rendered.request,
+ rendered.template_token_ids,
+ {"text": reply_text, "token_ids": reply_ids, "finish_reason": "stop"},
+ )
+ finally:
+ pool.shutdown()
+ return info, rendered, reply
+
+ info, rendered, reply = asyncio.run(serve())
+
+ # Turns end on <|im_end|>, which the splice in the frontend cuts at.
+ assert tokenizer.convert_tokens_to_ids("<|im_end|>") in info["turn_end_token_ids"]
+ assert info["max_model_len"] == MAX_MODEL_LEN
+ # The call, and the history up to its last assistant turn the splice cuts against.
+ assert rendered.template_token_ids == _template_ids(
+ tokenizer, MESSAGES, add_generation_prompt=True
+ )
+ assert rendered.history_token_ids == _template_ids(
+ tokenizer, MESSAGES[:2], add_generation_prompt=False
+ )
+ assert rendered.required_prefix_token_ids == [1, 2, 3]
+ assert rendered.request_max_tokens == 64
+ assert "messages" not in rendered.request
+ assert (rendered.admission, rendered.multimodal) == (None, None)
+ # The reply through vLLM's own reasoning and tool parsers.
+ (choice,) = reply["choices"]
+ assert choice["finish_reason"] == "tool_calls"
+ assert choice["message"]["reasoning"].strip() == "checking"
+ (tool_call,) = choice["message"]["tool_calls"]
+ assert tool_call["function"]["name"] == "get_weather"
+ assert json.loads(tool_call["function"]["arguments"]) == {"city": "Lyon"}
+
+
+def test_workers_start_before_their_settings_and_build_when_they_arrive(
+ tiny_qwen3_model_path,
+) -> None:
+ async def serve():
+ pool = RenderPool(workers=2)
+ try:
+ start = asyncio.ensure_future(pool.start())
+ # Both come up before any settings exist, as at the frontend's launch.
+ async with asyncio.timeout(60):
+ while len(pool._executor._processes) < 2:
+ await asyncio.sleep(0.01)
+ assert not start.done()
+ pool.build(SPEC, tiny_qwen3_model_path)
+ await start
+ return (await pool.info())["max_model_len"]
+ finally:
+ pool.shutdown()
+
+ assert asyncio.run(serve()) == MAX_MODEL_LEN
+
+
+def test_a_worker_waits_for_its_settings_until_its_pool_is_shut_down(tmp_path) -> None:
+ path = tmp_path / "settings.json"
+ with ThreadPoolExecutor(max_workers=1) as worker:
+ waiting = worker.submit(_wait_for_settings, str(path))
+ time.sleep(0.3)
+ assert not waiting.done()
+ path.write_text(json.dumps({"spec": SPEC, "model_dir": "/snapshots/model"}))
+ assert waiting.result(timeout=10) == (SPEC, "/snapshots/model")
+ # Shutdown removes the pool's directory; a worker left waiting would leak.
+ with pytest.raises(SystemExit):
+ _wait_for_settings(str(tmp_path / "removed" / "settings.json"))
+
+
+def test_a_dead_worker_fails_only_its_calls(tiny_qwen3_model_path) -> None:
+ body = {
+ "model": "model",
+ "messages": MESSAGES[:1],
+ "max_tokens": 8,
+ "temperature": 0.7,
+ "top_p": 0.9,
+ }
+
+ async def serve():
+ pool = _pool(SPEC, tiny_qwen3_model_path)
+ try:
+ await pool.info()
+ (pid,) = pool._executor._processes
+ # As the OOM killer would.
+ os.kill(pid, signal.SIGKILL)
+ with pytest.raises(BrokenProcessPool):
+ await pool.render(body)
+ # The next call gets a fresh pool, not the broken one.
+ return await pool.render(body)
+ finally:
+ pool.shutdown()
+
+ rendered = asyncio.run(serve())
+
+ assert rendered.template_token_ids
+
+
+def test_workers_that_cannot_start_fail_the_call_by_name(tiny_qwen3_model_path):
+ vllm_cfg = {"http_server_serving_chat_kwargs": {"reasoning_parser": "no_such"}}
+ spec = {**SPEC, "generation": {**SPEC["generation"], "vllm_cfg": vllm_cfg}}
+
+ async def serve():
+ pool = _pool(spec, tiny_qwen3_model_path)
+ try:
+ await pool.info()
+ finally:
+ pool.shutdown()
+
+ with pytest.raises(RuntimeError, match="before serving a call"):
+ asyncio.run(serve())
+
+
+def _nemotron_vl_loader() -> str:
+ """Run in a render worker: the class vLLM decodes nemotron_vl videos with."""
+ from vllm.multimodal.video import VIDEO_LOADER_REGISTRY
+
+ return VIDEO_LOADER_REGISTRY.name2class["nemotron_vl"].__name__
+
+
+def test_workers_decode_videos_with_nemo_rls_loader(tiny_qwen3_model_path) -> None:
+ # The worker decodes a Gym call's videos, so it needs the engine's loader.
+ vllm_cfg = {
+ **SPEC["generation"]["vllm_cfg"],
+ "video": {
+ "sampling_style": "nemotron_vl",
+ "num_frames": 8,
+ "temporal_patch_size": 2,
+ },
+ }
+ spec = {**SPEC, "generation": {**SPEC["generation"], "vllm_cfg": vllm_cfg}}
+
+ async def serve():
+ pool = _pool(spec, tiny_qwen3_model_path)
+ try:
+ return await pool._submit(_nemotron_vl_loader)
+ finally:
+ pool.shutdown()
+
+ assert asyncio.run(serve()) == "TorchCodecNemotronVLVideoBackend"
+
+
+class _RecordingParser:
+ """vLLM's response parser: the content is the text, and the IDs are kept."""
+
+ def __init__(self) -> None:
+ self.token_ids: list[int] | None = None
+
+ def parse(
+ self, model_output, request, *, enable_auto_tools, model_output_token_ids
+ ):
+ self.token_ids = list(model_output_token_ids)
+ return None, model_output, []
+
+ def count_reasoning_tokens(self, token_ids) -> int:
+ return 0
+
+ def set_prompt_token_ids(self, prompt_token_ids) -> None:
+ # vLLM hands the parser the prompt before it parses the reply.
+ pass
+
+
+@pytest.fixture(scope="module")
+def reply_worker(tiny_qwen3_model_path) -> _Worker:
+ """One render worker, in this process, to build replies with."""
+ return _Worker(SPEC, tiny_qwen3_model_path)
+
+
+@pytest.mark.parametrize("style", ["metaspace", "byte_level"])
+@pytest.mark.parametrize("case", RESTORE_CASES, ids=lambda case: case.name)
+def test_a_resumed_calls_reply_matches_the_uninterrupted_reply(
+ reply_worker, monkeypatch, case: RestoreCase, style: str
+) -> None:
+ """The cases the native server passes in test_prefix_restore_cases.py."""
+ tokenizer = _tokenizer(style)
+ parser = _RecordingParser()
+ monkeypatch.setattr(reply_worker, "tokenizer", tokenizer)
+ # The serving chat's reply step decodes the saved tokens with its renderer's.
+ monkeypatch.setattr(reply_worker.serving_chat.renderer, "tokenizer", tokenizer)
+ monkeypatch.setattr(reply_worker, "_parser", lambda chat: parser)
+ prompt = _ids(tokenizer, case.prompt)
+ output = _ids(tokenizer, case.output)
+ saved, tail = output[: case.saved], output[case.saved :]
+ # The engine ran the saved rows as its prompt and decoded only the tail.
+ terminal = _classify_restored_prefix_terminal(
+ prompt_token_ids=prompt + saved,
+ prefix_token_count=len(saved),
+ requested_output_tokens=len(output) + 8,
+ model_max_tokens=4096,
+ terminal_finish_reason=case.finish_reason if not tail else None,
+ terminal_stop_reason=case.stop_reason if not tail else None,
+ )
+ tail_text = ""
+ if tail:
+ tail_text = _engine_text(
+ tokenizer,
+ prompt + saved,
+ tail,
+ case,
+ min_tokens=max(0, case.min_tokens - len(saved)),
+ )
+
+ reply = reply_worker.respond(
+ {
+ "model": "model",
+ "min_tokens": case.min_tokens,
+ "stop": list(case.stop) or None,
+ "include_stop_str_in_output": case.include_stop_str_in_output,
+ },
+ # The prompt the frontend sent, which the engine swapped.
+ prompt,
+ {
+ "text": tail_text,
+ "token_ids": tail,
+ "finish_reason": case.finish_reason,
+ "stop_reason": case.stop_reason,
+ },
+ {
+ "prompt_token_ids": prompt + saved,
+ "token_ids": saved,
+ "terminal": None
+ if terminal is None
+ else {
+ "prefix_token_count": terminal.prefix_token_count,
+ "reason": terminal.reason,
+ "finish_reason": terminal.finish_reason,
+ "stop_reason": terminal.stop_reason,
+ },
+ },
+ )
+
+ (choice,) = reply["choices"]
+ assert choice["message"]["content"] == _engine_text(
+ tokenizer, prompt, output, case, min_tokens=case.min_tokens
+ )
+ assert choice["finish_reason"] == case.finish_reason
+ # Tool-call and reasoning parsers see the whole output, saved tokens first.
+ assert parser.token_ids == output
+ usage = reply["usage"]
+ assert (usage["prompt_tokens"], usage["completion_tokens"]) == (
+ len(prompt),
+ len(output),
+ )
diff --git a/tests/unit/models/generation/test_dynamo_vllm_adapter.py b/tests/unit/models/generation/test_dynamo_vllm_adapter.py
index 04ba37a7c77..bdb3b0ff96c 100644
--- a/tests/unit/models/generation/test_dynamo_vllm_adapter.py
+++ b/tests/unit/models/generation/test_dynamo_vllm_adapter.py
@@ -14,14 +14,16 @@
"""The vLLM Dynamo adapter over a fake AsyncLLM.
The host is a real ``VllmAsyncGenerationWorkerImpl`` built without its
-constructor, so rollouts are sampled by the native worker's own helpers.
+constructor, so rollouts and NeMo-Gym calls run the native worker's own helpers.
"""
import ast
import asyncio
+import base64
import importlib.util
import json
import logging
+import pickle
from pathlib import Path
from types import SimpleNamespace
@@ -34,11 +36,9 @@
vllm = pytest.importorskip("vllm")
from dynamo.llm.exceptions import InvalidArgument # noqa: E402
+from vllm.multimodal.inputs import PlaceholderRange # noqa: E402
-from nemo_rl.models.generation.dynamo import ( # noqa: E402
- engine_adapter,
- vllm_adapter,
-)
+from nemo_rl.models.generation.dynamo import engine_adapter # noqa: E402
from nemo_rl.models.generation.dynamo.config import ( # noqa: E402
DynamoCfg,
DynamoFrontendArgs,
@@ -46,45 +46,67 @@
from nemo_rl.models.generation.dynamo.rollout_client import ( # noqa: E402
ROLLOUT_ANNOTATION,
)
+from nemo_rl.models.generation.dynamo.vllm import ( # noqa: E402
+ adapter as vllm_adapter,
+)
+from nemo_rl.models.generation.dynamo.vllm.chat_engine import ( # noqa: E402
+ CALL_KEY,
+ GYM_CALL_ANNOTATION,
+ RUNTIME_DATA_KEY,
+ ForwardedCall,
+)
+from nemo_rl.models.generation.interfaces import SamplerReplayConfig # noqa: E402
from nemo_rl.models.generation.rollout_result import ( # noqa: E402
rollout_result_from_wire,
rollout_result_to_wire,
)
+from nemo_rl.models.generation.vllm.chat_server import ( # noqa: E402
+ NeMoRLChatCompletionRequest,
+)
from nemo_rl.models.generation.vllm.request_pipeline import ( # noqa: E402
rollout_output,
rollout_result,
)
+from nemo_rl.models.generation.vllm.utils import R3_MISSING_ROUTE_SENTINEL # noqa: E402
from nemo_rl.models.generation.vllm.vllm_worker_async import ( # noqa: E402
VllmAsyncGenerationWorkerImpl,
)
+from nemo_rl.utils.routed_experts_codec import decode_routed_experts # noqa: E402
MAX_MODEL_LEN = 16
+EOS = 2
+# A processed image's placeholder tokens.
+IMAGE_TOKEN, IMAGE_TOKENS = 91, 3
class _Context:
def id(self) -> str:
return "request-0"
+ def is_stopped(self) -> bool:
+ return False
+
-def _logprob(value: float):
- return SimpleNamespace(logprob=value, rank=1, decoded_token=None)
+def _logprob(value: float, rank: int = 1):
+ return SimpleNamespace(logprob=value, rank=rank, decoded_token=None)
def _completion(token_ids, finish_reason=None, index=0, routed_experts=None):
return SimpleNamespace(
index=index,
+ text="".join(f"<{token_id}>" for token_id in token_ids),
token_ids=token_ids,
logprobs=[{t: _logprob(-0.5)} for t in token_ids],
finish_reason=finish_reason,
+ stop_reason=None,
routed_experts=routed_experts,
)
-def _output(prompt, *completions, prompt_routed_experts=None, prompt_logprobs=None):
+def _output(prompt, *completions, prompt_routed_experts=None):
return SimpleNamespace(
prompt_token_ids=prompt,
num_cached_tokens=0,
- prompt_logprobs=prompt_logprobs,
prompt_routed_experts=prompt_routed_experts,
outputs=list(completions),
)
@@ -93,29 +115,44 @@ def _output(prompt, *completions, prompt_routed_experts=None, prompt_logprobs=No
class _FakeLLM:
"""Yields scripted RequestOutputs and records what it was asked to generate."""
- def __init__(self, outputs=()):
+ def __init__(self, outputs=(), *, defaults=None, admission_error=None):
self.outputs = list(outputs)
self.calls = []
+ self.admission_error = admission_error
self.tokenizer = None
+ self.model_config = SimpleNamespace(
+ get_diff_sampling_param=lambda: dict(defaults or {}),
+ try_get_generation_config=lambda: {},
+ generation_config="auto",
+ override_generation_config={},
+ )
self.vllm_config = SimpleNamespace(
model_config=SimpleNamespace(max_model_len=MAX_MODEL_LEN),
kv_events_config=None,
)
- async def generate(self, prompt, sampling_params, request_id):
- self.calls.append((prompt, sampling_params, request_id))
+ def check_admission(self, n, request_id):
+ if self.admission_error is not None:
+ raise self.admission_error
+
+ async def generate(self, prompt, sampling_params, request_id, **kwargs):
+ self.calls.append((prompt, sampling_params, request_id, kwargs))
for output in self.outputs:
yield output
def _host(llm: _FakeLLM, **cfg) -> VllmAsyncGenerationWorkerImpl:
host = object.__new__(VllmAsyncGenerationWorkerImpl)
+ host._init_token_capture_state()
host.cfg = {
"max_new_tokens": 8,
"temperature": 0.7,
"top_p": 0.9,
+ "val_temperature": 0.0,
+ "val_top_p": 1.0,
"top_k": None,
- "stop_token_ids": [2],
+ "val_top_k": None,
+ "stop_token_ids": [EOS],
"stop_strings": [""],
"vllm_cfg": {"max_model_len": MAX_MODEL_LEN},
**cfg,
@@ -124,12 +161,18 @@ def _host(llm: _FakeLLM, **cfg) -> VllmAsyncGenerationWorkerImpl:
host.model_name = "model"
host.routed_experts_dtype = torch.int16
host.llm = llm
+ host._http_engine_client = None
+ host._capture_media = False
+ host._capture_top_logprobs = 0
+ host.token_capture = None
return host
-def _collect(engine, request) -> list[dict]:
+def _collect(engine, request, context=None) -> list[dict]:
async def run():
- return [chunk async for chunk in engine.generate(request, _Context())]
+ return [
+ chunk async for chunk in engine.generate(request, context or _Context())
+ ]
return asyncio.run(run())
@@ -149,7 +192,7 @@ def test_rollouts_are_served_like_generate_async() -> None:
engine, _rollout_request([1, 2, 3], greedy=True, stop_strings=[""])
)
- ((_, sampling_params, _),) = llm.calls
+ ((_, sampling_params, _, _),) = llm.calls
# The worker's _serve_rollout samples it, as it does for generate_async...
assert sampling_params.max_tokens == 8
assert sampling_params.temperature == 0.0
@@ -183,7 +226,7 @@ async def abort(request_id):
# Dynamo's worker calls abort() when the frontend cancels a request.
asyncio.run(engine.abort(_Context()))
- ((_, _, request_id),) = llm.calls
+ ((_, _, request_id, _),) = llm.calls
assert aborted == [request_id]
@@ -274,7 +317,7 @@ async def start():
return config
-def test_engines_register_for_rollouts_on_completions_only(monkeypatch) -> None:
+def test_engines_register_for_rollouts_and_nemo_gym_chat(monkeypatch) -> None:
config = _start_worker_config(monkeypatch, _host(_FakeLLM()))
assert config.namespace == "ns"
@@ -283,8 +326,8 @@ def test_engines_register_for_rollouts_on_completions_only(monkeypatch) -> None:
"/snapshots/model",
"model",
)
- # NeMo RL's rollouts go to /v1/completions; no chat surface is registered.
- assert config.endpoint_types == "completions"
+ # NeMo RL's rollouts go to /v1/completions, NeMo-Gym's calls to chat.
+ assert config.endpoint_types == "chat,completions"
# Worker.run() starts with Dynamo's logging setup, which keeps the worker's.
handlers = logging.getLogger().handlers[:]
engine_adapter.dynamo_worker.configure_dynamo_logging()
@@ -298,11 +341,11 @@ def test_worker_run_still_calls_the_logging_setup_the_engine_replaces() -> None:
assert "configure_dynamo_logging" in run.__code__.co_names
-def test_requests_other_than_rollouts_are_refused() -> None:
+def test_requests_other_than_nemo_rls_are_refused() -> None:
llm = _FakeLLM()
engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
- with pytest.raises(InvalidArgument, match="only NeMo RL rollouts"):
+ with pytest.raises(InvalidArgument, match="only NeMo RL's own requests"):
_collect(engine, {"token_ids": [1, 2, 3, 4]})
assert llm.calls == []
@@ -359,15 +402,309 @@ async def call_utility_async(method):
]
llm.engine_core = SimpleNamespace(call_utility_async=call_utility_async)
- engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
+ host = _host(llm)
+ host.llm_async_engine_args = vllm.AsyncEngineArgs(model="/models/m")
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
- registration = asyncio.run(engine.start(worker_id=0)).llm
+ config = asyncio.run(engine.start(worker_id=0))
+ registration = config.llm
# The KV router hashes events in blocks of the main-attention group; a
# wrong size would quietly turn KV routing into load-only routing.
assert registration.kv_cache_block_size == 528
assert registration.total_kv_blocks == 64
assert registration.context_length == MAX_MODEL_LEN
+ # The frontend's chat engine renders with what the engine publishes.
+ assert config.runtime_data == vllm_adapter.chat_runtime_data(host)
+
+
+def test_an_engine_sends_its_chat_settings_before_it_builds_the_engine(
+ monkeypatch,
+) -> None:
+ host = _host(_FakeLLM())
+ sent: list[dict] = []
+ host.announce_dynamo_chat_settings_to(SimpleNamespace(put=sent.append))
+ sent_before_the_build: list[list[dict]] = []
+
+ def build(engine_args, stat_loggers):
+ sent_before_the_build.append(list(sent))
+ return _FakeLLM()
+
+ monkeypatch.setattr("vllm.v1.engine.async_llm.AsyncLLM.from_engine_args", build)
+
+ host._create_engine({"model": "/models/m", "max_model_len": MAX_MODEL_LEN})
+
+ # What it publishes at registration, while the frontend can still build its
+ # render workers ahead of it.
+ assert sent_before_the_build == [[vllm_adapter.chat_runtime_data(host)]]
+
+
+def test_engines_publish_what_the_frontends_chat_engine_renders_with() -> None:
+ host = _host(_FakeLLM())
+ host.llm_async_engine_args = vllm.AsyncEngineArgs(
+ model="/models/m",
+ max_model_len=MAX_MODEL_LEN,
+ enable_prefix_caching=True,
+ skip_tokenizer_init=True,
+ )
+ serving_chat_kwargs = {"tool_parser": "hermes", "enable_auto_tools": True}
+ host.cfg["vllm_cfg"] = {
+ "max_model_len": MAX_MODEL_LEN,
+ "http_server_serving_chat_kwargs": serving_chat_kwargs,
+ "reasoning_parser_plugin": "/plugins/reasoning.py",
+ }
+ # MasterConfig validates this block into a pydantic model.
+ host.cfg["sampler_replay"] = SamplerReplayConfig(enabled=True)
+
+ # Dynamo carries runtime_data as JSON.
+ spec = json.loads(json.dumps(vllm_adapter.chat_runtime_data(host)))[
+ RUNTIME_DATA_KEY
+ ]
+
+ # The engine's own args rebuild the same model config, tokenizer included.
+ published = vllm.AsyncEngineArgs(**spec["engine_args"])
+ for field in ("model", "max_model_len", "enable_prefix_caching"):
+ assert getattr(published, field) == getattr(
+ host.llm_async_engine_args, field
+ ), field
+ assert published.skip_tokenizer_init is False
+ # All of policy.generation, which the native server's chat helpers read.
+ assert spec["generation"] == {**host.cfg, "sampler_replay": {"enabled": True}}
+
+
+# --- NeMo-Gym calls, after the frontend's chat engine rendered them -----------
+
+
+def _chat(**fields) -> dict:
+ """A rendered chat request, as the chat engine sends it (without messages)."""
+ return NeMoRLChatCompletionRequest(
+ model="model",
+ messages=[{"role": "user", "content": "hi"}],
+ temperature=0.7,
+ top_p=0.9,
+ **fields,
+ ).model_dump(mode="json", exclude={"messages"})
+
+
+def _gym_call(token_ids: list[int], **fields) -> dict:
+ """A Gym call as the chat engine routes it: final token ids and the plan.
+
+ ``fields`` set the prepared prompt's fields and the call's others by name.
+ """
+ plan = {
+ "splice": None,
+ "prefix_len": None,
+ "request_max_tokens": None,
+ "max_tokens": None,
+ "admission": None,
+ "serving_admission": None,
+ "verify_retained_media": False,
+ }
+ wire: ForwardedCall = {
+ "prepared_prompt": plan,
+ "request": _chat(),
+ "multimodal": None,
+ "reasoning_ended": None,
+ "reasoning_parser_kwargs": None,
+ }
+ # A media call's template ids are its own prompt's unless it was spliced.
+ template_token_ids = fields.pop("template_token_ids", token_ids)
+ if (inputs := fields.pop("multimodal", None)) is not None:
+ wire["multimodal"] = {
+ "inputs": inputs,
+ "template_token_ids": template_token_ids,
+ }
+ for name, value in fields.items():
+ if name in plan:
+ plan[name] = value
+ else:
+ assert name in wire, name
+ wire[name] = value
+ return {
+ "token_ids": token_ids,
+ "annotations": [GYM_CALL_ANNOTATION],
+ "extra_args": {CALL_KEY: wire},
+ }
+
+
+@pytest.mark.parametrize(
+ ("max_tokens", "served_max_tokens"), [(5, 5), (None, MAX_MODEL_LEN - 3)]
+)
+def test_gym_calls_generate_with_the_native_servers_sampling_params(
+ max_tokens, served_max_tokens
+) -> None:
+ final = _output([1, 3, 4], _completion([10, 11], "stop"))
+ llm = _FakeLLM([final])
+ engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
+ reasoning = {"chat_template_kwargs": {"enable_thinking": True}}
+
+ (chunk,) = _collect(
+ engine,
+ _gym_call(
+ [1, 3, 4],
+ # Gym's token information: logprobs=true, top_logprobs=0.
+ request=_chat(
+ max_tokens=max_tokens, stop=[""], logprobs=True, top_logprobs=0
+ ),
+ reasoning_ended=False,
+ reasoning_parser_kwargs=reasoning,
+ ),
+ )
+
+ ((prompt, params, _, kwargs),) = llm.calls
+ assert prompt == {"prompt_token_ids": [1, 3, 4]}
+ # The request's own sampling, as vLLM's chat server builds it.
+ assert (params.temperature, params.top_p) == (0.7, 0.9)
+ assert (params.max_tokens, params.stop, params.logprobs) == (
+ served_max_tokens,
+ [""],
+ 0,
+ )
+ assert params.output_kind == vllm.sampling_params.RequestOutputKind.FINAL_ONLY
+ # What vLLM's chat serving hands the engine besides the prompt.
+ assert kwargs == {"reasoning_ended": False, "reasoning_parser_kwargs": reasoning}
+ # The output the frontend builds the reply from, and the call's result.
+ assert chunk["engine_data"] == {
+ "output": {
+ "text": "<10><11>",
+ "token_ids": [10, 11],
+ "finish_reason": "stop",
+ "stop_reason": None,
+ "logprobs": [{"10": [-0.5, 1, None]}, {"11": [-0.5, 1, None]}],
+ },
+ "result": {
+ "prompt_token_ids": [1, 3, 4],
+ "generation_token_ids": [10, 11],
+ "generation_log_probs": [-0.5, -0.5],
+ },
+ }
+ assert chunk["completion_usage"]["prompt_tokens"] == 3
+
+
+def test_an_uncaptured_gym_call_carries_its_routes() -> None:
+ # vLLM routes every token but the last sampled one.
+ routes = np.arange(4 * 2 * 2, dtype=np.int16).reshape(4, 2, 2)
+ final = _output(
+ [1, 3, 4],
+ _completion([10, 11], "stop", routed_experts=routes[3:]),
+ prompt_routed_experts=routes[:3],
+ )
+ host = _host(_FakeLLM([final]), vllm_kwargs={"enable_return_routed_experts": True})
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ (chunk,) = _collect(engine, _gym_call([1, 3, 4]))
+
+ # As the native endpoint attaches them: one row per token of the sequence.
+ expected = torch.full((5, 2, 2), R3_MISSING_ROUTE_SENTINEL, dtype=torch.int16)
+ expected[:4] = torch.from_numpy(routes)
+ shipped = chunk["engine_data"]["result"]["routed_experts"]
+ assert torch.equal(decode_routed_experts(shipped, dtype=None), expected)
+
+
+def test_a_gym_call_without_routes_fails_under_router_replay() -> None:
+ final = _output([1, 3, 4], _completion([10, 11], "stop"))
+ host = _host(_FakeLLM([final]), vllm_kwargs={"enable_return_routed_experts": True})
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ # Training would replay the router on routes the call never returned.
+ with pytest.raises(InvalidArgument, match="did not include routed_experts"):
+ _collect(engine, _gym_call([1, 3, 4]))
+
+
+def test_an_uncaptured_gym_call_with_non_finite_logprobs_fails_as_natively() -> None:
+ completion = _completion([10, 11], "length")
+ completion.logprobs[1] = {11: _logprob(float("nan"))}
+ llm = _FakeLLM([_output([1, 3, 4], completion)])
+ engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
+
+ # Dynamo would hand the frontend a null log-prob. Fail the call instead: the
+ # chat engine answers an engine's error with 500, as natively, and Gym retries.
+ with pytest.raises(
+ InvalidArgument, match=r"non-finite log-prob \(nan\) at token 1"
+ ):
+ _collect(engine, _gym_call([1, 3, 4]))
+
+
+def test_a_gym_call_without_the_chat_engines_plan_is_refused() -> None:
+ llm = _FakeLLM()
+ engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
+
+ with pytest.raises(InvalidArgument, match=r"extra_args\.nemo_rl"):
+ _collect(engine, {"token_ids": [1, 3, 4], "annotations": [GYM_CALL_ANNOTATION]})
+
+ assert llm.calls == []
+
+
+@pytest.mark.parametrize(
+ ("chat", "admission_error", "status", "acknowledge", "message"),
+ [
+ # vLLM's own validation errors keep their 400 and message, as natively,
+ # and acknowledge a capture call's refusal.
+ (_chat(min_tokens=10, max_tokens=5), None, 400, True, "min_tokens"),
+ # The queue limits the native server checks: a 503 to retry elsewhere.
+ (_chat(), vllm.exceptions.QueueOverflowError(), 503, False, "busy"),
+ ],
+ ids=["request_error", "queue_full"],
+)
+def test_a_gym_call_the_engine_refuses_answers_with_the_native_status(
+ chat, admission_error, status, acknowledge, message
+) -> None:
+ llm = _FakeLLM(
+ [_output([1, 3, 4], _completion([10], "stop"))],
+ admission_error=admission_error,
+ )
+ engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
+
+ # In the chunk: Dynamo hands a raised error to the chat engine without it.
+ (chunk,) = _collect(engine, _gym_call([1, 3, 4], request=chat))
+
+ refusal = chunk["engine_data"]["refusal"]
+ assert (refusal["status"], refusal["acknowledge"]) == (status, acknowledge)
+ assert message in refusal["message"]
+ assert llm.calls == []
+
+
+def test_a_spliced_media_call_keeps_placeholders_on_their_tokens() -> None:
+ # Turn 1 ran [1, , 3] and generated [5, 6, EOS]; the template
+ # re-rendered that turn as [7, EOS] and appended a turn with a second image.
+ image = [IMAGE_TOKEN] * IMAGE_TOKENS
+ template = [1, *image, 3, 7, EOS, *image, 4]
+ prefix = [1, *image, 3, 5, 6, EOS]
+ spliced = [*prefix, *image, 4]
+ multimodal = {
+ "type": "multimodal",
+ "mm_kwargs": {"image": ["pixels-0", "pixels-1"]},
+ "mm_hashes": {"image": ["h0", "h1"]},
+ "mm_placeholders": {
+ "image": [
+ PlaceholderRange(1, IMAGE_TOKENS),
+ PlaceholderRange(7, IMAGE_TOKENS),
+ ]
+ },
+ }
+ llm = _FakeLLM([_output(spliced, _completion([10], "stop"))])
+ engine = vllm_adapter.DynamoVllmAdapter(_host(llm), leader_worker_idx=0)
+
+ _collect(
+ engine,
+ _gym_call(
+ spliced,
+ template_token_ids=template,
+ # Each side cut at its turn end: prefix[:7] + template[6:].
+ splice=[7, 6],
+ multimodal=base64.b64encode(pickle.dumps(multimodal)).decode(),
+ ),
+ )
+
+ ((engine_input, params, _, _),) = llm.calls
+ assert engine_input["prompt_token_ids"] == spliced
+ assert engine_input["mm_kwargs"] == multimodal["mm_kwargs"]
+ # The first image sits in the model's own prefix; the second moved with the
+ # longer generated turn, as the native endpoint remaps them.
+ assert [
+ (span.offset, span.length) for span in engine_input["mm_placeholders"]["image"]
+ ] == [(1, IMAGE_TOKENS), (len(prefix), IMAGE_TOKENS)]
+ assert params.max_tokens == MAX_MODEL_LEN - len(spliced)
@pytest.mark.parametrize(
diff --git a/tests/unit/models/generation/test_dynamo_vllm_adapter_capture.py b/tests/unit/models/generation/test_dynamo_vllm_adapter_capture.py
new file mode 100644
index 00000000000..50c869c43df
--- /dev/null
+++ b/tests/unit/models/generation/test_dynamo_vllm_adapter_capture.py
@@ -0,0 +1,582 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""NeMo-Gym token capture through the Dynamo engine adapter.
+
+The frontend's chat engine rendered and spliced each call; the adapter runs the
+worker's own capture lifecycle on it, as the native chat endpoint does. Only
+Gym's capture core, and in the media tests the worker's media snapshot, are
+faked.
+"""
+
+import base64
+import pickle
+import time
+from types import SimpleNamespace
+from unittest.mock import MagicMock
+
+import numpy as np
+import pytest
+import torch
+
+pytestmark = [pytest.mark.nemo_gym, pytest.mark.vllm]
+pytest.importorskip("dynamo._core")
+pytest.importorskip("vllm")
+# The vLLM lanes import this module without the nemo_gym extra.
+pytest.importorskip("nemo_gym.token_id_capture.staging")
+
+from dynamo.llm.exceptions import InvalidArgument # noqa: E402
+from nemo_gym._checkpoint.generation_cut import ( # noqa: E402
+ GenerationCutInventory,
+ GenerationCutPrefix,
+)
+from nemo_gym.token_id_capture import NG_COMMIT_COORDS_FIELD # noqa: E402
+from nemo_gym.token_id_capture.adapters.vllm import ( # noqa: E402
+ MEDIA_SPANS_FIELD,
+ VLLMCaptureAdapter,
+)
+from nemo_gym.token_id_capture.replay import ( # noqa: E402
+ replay_context,
+ summarize_replay,
+)
+from nemo_gym.token_id_capture.staging.capture import ( # noqa: E402
+ RolloutTokenCapture,
+)
+from nemo_gym.token_id_capture.staging.records import CaptureAdmission # noqa: E402
+from vllm.multimodal.inputs import PlaceholderRange # noqa: E402
+
+from nemo_rl.data.captured_media import ( # noqa: E402
+ MEDIA_TENSOR_DIGESTS_FIELD,
+ CapturedMedia,
+)
+from nemo_rl.data_plane.tq_token_sink import ChainPrefixCache # noqa: E402
+from nemo_rl.models.generation.dynamo.vllm import ( # noqa: E402
+ adapter as vllm_adapter,
+)
+from nemo_rl.models.generation.dynamo.vllm.generation_cut import ( # noqa: E402
+ CUT_PREFIXES_KEY,
+ CUT_REPLY_MARGIN_S,
+ DEADLINE_KEY,
+ GENERATION_CUT_ANNOTATION,
+ INVENTORY_KEY,
+)
+from nemo_rl.models.generation.prefix_recovery import NoPrefixRecovery # noqa: E402
+from nemo_rl.models.generation.vllm.utils import R3_MISSING_ROUTE_SENTINEL # noqa: E402
+from nemo_rl.utils.routed_experts_codec import ( # noqa: E402
+ routed_experts_tensor_metadata,
+)
+from tests.unit.models.generation.test_dynamo_vllm_adapter import ( # noqa: E402
+ EOS,
+ IMAGE_TOKEN,
+ IMAGE_TOKENS,
+ _chat,
+ _collect,
+ _completion,
+ _Context,
+ _FakeLLM,
+ _gym_call,
+ _host,
+ _logprob,
+ _output,
+)
+from tests.unit.models.generation.test_vllm_token_capture_hosting import ( # noqa: E402
+ _active_call_inventory,
+ _assert_stages_captured_tensors,
+ _captured_image_media,
+ _cuts,
+ _live_image_call,
+ _MemorySink,
+ _single_call_inventory,
+)
+
+COORDS = {"staging_key": "rollout-1/c2"}
+# Turn 2 of a rollout: the staged prefix [1, 5, 6, EOS] replaces the template's
+# re-render of turn 1, [1, 7, EOS], before the new turn [3, 4].
+PREFIX = [1, 5, 6, EOS]
+TEMPLATE = [1, 7, EOS, 3, 4]
+# Each side cut at its turn end: PREFIX[:3] + TEMPLATE[2:].
+SPLICE = [3, 2]
+SPLICED = [*PREFIX, 3, 4]
+TOKEN_IN = CaptureAdmission(
+ capture_key="rollout-1",
+ model_call_id="c2",
+ parent_call_id="c1",
+ prev_len=len(PREFIX),
+ mode="token_in",
+ required_prefix_token_ids=PREFIX,
+ parent_chain_hash="0" * 64,
+)
+TEXT = CaptureAdmission(capture_key="rollout-1", model_call_id="c1", mode="text")
+
+
+class _Capture:
+ """Gym's capture core: records begin/complete/fail."""
+
+ adapter = VLLMCaptureAdapter()
+
+ def __init__(self) -> None:
+ self.begun: list[tuple] = []
+ self.staged: list[dict] = []
+ self.attachments: list[dict | None] = []
+ self.failed: list[str] = []
+
+ def begin_call(
+ self,
+ admission,
+ *,
+ prefix_token_ids,
+ generation_cut,
+ generation_cut_staging_keys,
+ stream,
+ ):
+ del generation_cut_staging_keys
+ self.begun.append((admission, prefix_token_ids, stream))
+ return SimpleNamespace(
+ admission=admission,
+ model_call_id=admission.model_call_id,
+ capture_key=admission.capture_key,
+ generation_cut=generation_cut,
+ )
+
+ def complete_call_from_response(self, call, payload, attachments=None):
+ self.staged.append(payload)
+ self.attachments.append(attachments)
+ return SimpleNamespace(model_dump=lambda: dict(COORDS))
+
+ def fail_call(self, call, *, reason) -> None:
+ self.failed.append(reason)
+
+
+def _capturing_host(llm, **cfg):
+ host = _host(llm, **cfg)
+ host.token_capture = _Capture()
+ host._chain_prefix = ChainPrefixCache()
+ return host
+
+
+def _captured_call(
+ token_ids: list[int], admission: CaptureAdmission, top_logprobs: int = 0, **plan
+) -> dict:
+ """A captured Gym call, as the chat engine routes it after its render."""
+ admission_json = admission.model_dump(mode="json")
+ return _gym_call(
+ token_ids,
+ # Gym's dynamo_worker handler asks for the sampled tokens' logprobs.
+ request=_chat(
+ logprobs=True, top_logprobs=top_logprobs, ng_capture=admission_json
+ ),
+ admission=admission_json,
+ serving_admission=admission_json,
+ **plan,
+ )
+
+
+def test_a_captured_token_in_call_is_staged_from_the_engine() -> None:
+ # vLLM routes every token but the last sampled one.
+ routes = np.arange(7 * 2 * 2, dtype=np.int16).reshape(7, 2, 2)
+ final = _output(
+ SPLICED,
+ _completion([10, 11], "stop", routed_experts=routes[6:]),
+ prompt_routed_experts=routes[:6],
+ )
+ llm = _FakeLLM([final])
+ host = _capturing_host(llm, vllm_kwargs={"enable_return_routed_experts": True})
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ (chunk,) = _collect(
+ engine,
+ _captured_call(
+ SPLICED,
+ TOKEN_IN,
+ prefix_len=len(PREFIX),
+ splice=SPLICE,
+ ),
+ )
+
+ ((prompt, params, _, _),) = llm.calls
+ assert prompt["prompt_token_ids"] == SPLICED
+ assert params.logprobs == 0
+ # The model's own prefix, as the frontend spliced it, begins the call.
+ assert host.token_capture.begun == [(TOKEN_IN, PREFIX, False)]
+ (payload,) = host.token_capture.staged
+ assert payload["prompt_token_ids"] == SPLICED
+ message = payload["choices"][0]["message"]
+ assert message["generation_token_ids"] == [10, 11]
+ assert message["generation_log_probs"] == [-0.5, -0.5]
+ # As the native endpoint stages them: the full sequence's routes (none for
+ # the last token) from the boundary row, the prefix's last token, on.
+ (attachments,) = host.token_capture.attachments
+ staged_routes = attachments["routed_experts"]
+ expected = torch.full((5, 2, 2), R3_MISSING_ROUTE_SENTINEL, dtype=torch.int16)
+ expected[:4] = torch.from_numpy(routes[3:7])
+ assert torch.equal(staged_routes, expected)
+ assert message["routed_experts"] == routed_experts_tensor_metadata(staged_routes)
+ # Only the commit coordinates go back through the frontend, beside the text.
+ assert chunk["engine_data"]["result"] == {NG_COMMIT_COORDS_FIELD: COORDS}
+ assert "logprobs" not in chunk["engine_data"]["output"]
+ assert len(host._capture_registry) == 0
+
+
+def test_a_captured_call_stages_the_samplers_top_k_for_score_centering() -> None:
+ completion = _completion([10, 11], "stop")
+ completion.logprobs = [
+ {10: _logprob(-0.5), 20: _logprob(-2.0, rank=2)},
+ {11: _logprob(-0.7), 21: _logprob(-3.0, rank=2)},
+ ]
+ llm = _FakeLLM([_output([1, 3, 4], completion)])
+ host = _capturing_host(llm)
+ host._capture_top_logprobs = 2
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ _collect(engine, _captured_call([1, 3, 4], TEXT, top_logprobs=2))
+
+ ((_, params, _, _),) = llm.calls
+ assert params.logprobs == 2
+ (attachments,) = host.token_capture.attachments
+ assert attachments["sampler_topk_token_ids"].tolist() == [[10, 20], [11, 21]]
+ assert torch.equal(
+ attachments["sampler_topk_logprobs"],
+ torch.tensor([[-0.5, -2.0], [-0.7, -3.0]], dtype=torch.float32),
+ )
+ assert host.token_capture.failed == []
+
+
+def test_an_undecided_candidate_admission_is_refused() -> None:
+ history = [{"role": "user", "content": "q"}, {"role": "assistant", "content": "a"}]
+ candidate = CaptureAdmission(
+ **{
+ **TOKEN_IN.model_dump(),
+ "mode": "candidate",
+ "requires_empty_tool_content_equivalence": False,
+ "candidate_replay": summarize_replay(
+ replay_context(history, render_digest="b" * 64)
+ ),
+ "request_replay": replay_context(history, render_digest="b" * 64),
+ }
+ )
+ llm = _FakeLLM()
+ host = _capturing_host(llm)
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ # The frontend's render decides a candidate; one reaching the engine skipped it.
+ with pytest.raises(InvalidArgument, match="candidate"):
+ _collect(engine, _captured_call(SPLICED, candidate))
+
+ assert (llm.calls, host.token_capture.begun) == ([], [])
+
+
+class _FailingLLM(_FakeLLM):
+ async def generate(self, prompt, sampling_params, request_id, **kwargs):
+ raise RuntimeError("engine died")
+ yield # pragma: no cover
+
+
+class _Disconnected(_Context):
+ def is_stopped(self) -> bool:
+ return True
+
+
+@pytest.mark.parametrize(
+ ("llm", "context", "error", "reason"),
+ [
+ (_FailingLLM(), _Context(), "engine died", "engine_error"),
+ # As vLLM's chat server: a call vLLM ended with an error is never staged.
+ (
+ _FakeLLM([_output([1, 3, 4], _completion([10], "error"))]),
+ _Context(),
+ "Generation failed",
+ "engine_error",
+ ),
+ # Dynamo aborts the vLLM request on a disconnect but lets the generator finish.
+ (
+ _FakeLLM([_output([1, 3, 4], _completion([10], "stop"))]),
+ _Disconnected(),
+ None,
+ "client_disconnect",
+ ),
+ ],
+ ids=["engine_error", "finished_with_error", "client_disconnect"],
+)
+def test_a_call_that_does_not_finish_cleanly_fails_its_capture(
+ llm, context, error, reason
+) -> None:
+ host = _capturing_host(llm)
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ if error is None:
+ assert _collect(engine, _captured_call([1, 3, 4], TEXT), context) == []
+ else:
+ with pytest.raises(InvalidArgument, match=error):
+ _collect(engine, _captured_call([1, 3, 4], TEXT), context)
+
+ assert host.token_capture.staged == []
+ assert host.token_capture.failed == [reason]
+ assert len(host._capture_registry) == 0
+
+
+def _multimodal(placeholders: list[PlaceholderRange]) -> str:
+ """The rendered engine input's media, as the chat engine ships it."""
+ multimodal = {
+ "type": "multimodal",
+ "mm_kwargs": {"image": [f"pixels-{i}" for i in range(len(placeholders))]},
+ "mm_hashes": {"image": [f"hash-{i}" for i in range(len(placeholders))]},
+ "mm_placeholders": {"image": placeholders},
+ }
+ return base64.b64encode(pickle.dumps(multimodal)).decode()
+
+
+def test_a_captured_media_call_keeps_the_placeholders_capture_moved() -> None:
+ image = [IMAGE_TOKEN] * IMAGE_TOKENS
+ template = [1, 7, EOS, *image, 4]
+ spliced = [*PREFIX, *image, 4]
+ llm = _FakeLLM([_output(spliced, _completion([10], "stop"))])
+ host = _capturing_host(llm)
+ host._capture_media = True
+ moved = {"image": [PlaceholderRange(len(PREFIX), IMAGE_TOKENS)]}
+ snapshots = []
+
+ def capture_request_media(
+ engine_input, *, admission, splice, verify_retained_media
+ ):
+ # As capture_processed_media, which moves the placeholders itself.
+ snapshots.append((admission, splice, verify_retained_media))
+ engine_input["mm_placeholders"] = moved
+ return CapturedMedia((), None)
+
+ host._capture_request_media = capture_request_media
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ _collect(
+ engine,
+ _captured_call(
+ spliced,
+ TOKEN_IN,
+ prefix_len=len(PREFIX),
+ template_token_ids=template,
+ splice=SPLICE,
+ multimodal=_multimodal([PlaceholderRange(3, IMAGE_TOKENS)]),
+ verify_retained_media=True,
+ ),
+ )
+
+ ((admission, splice, verify),) = snapshots
+ assert (admission, verify) == (TOKEN_IN, True)
+ assert (splice.token_ids, splice.model_cut_end, splice.template_cut_start) == (
+ spliced,
+ *SPLICE,
+ )
+ ((engine_input, _, _, _),) = llm.calls
+ assert engine_input["prompt_token_ids"] == spliced
+ # Not remapped a second time.
+ assert engine_input["mm_placeholders"] == moved
+
+
+def test_a_captured_media_call_stages_the_media_the_engine_ran_on() -> None:
+ processed = [1, *[IMAGE_TOKEN] * IMAGE_TOKENS, 3]
+ llm = _FakeLLM([_output(processed, _completion([10], "stop"))])
+ host = _capturing_host(llm)
+ host._capture_media = True
+ snapshots = []
+
+ def capture_request_media(
+ engine_input, *, admission, splice, verify_retained_media
+ ):
+ snapshots.append((list(engine_input["prompt_token_ids"]), admission, splice))
+ return CapturedMedia((), None)
+
+ host._capture_request_media = capture_request_media
+ engine = vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0)
+
+ (chunk,) = _collect(
+ engine,
+ _captured_call(
+ processed,
+ TEXT,
+ multimodal=_multimodal([PlaceholderRange(1, IMAGE_TOKENS)]),
+ ),
+ )
+
+ # Media is snapshotted from the processed prompt the engine runs.
+ assert snapshots == [(processed, TEXT, None)]
+ (payload,) = host.token_capture.staged
+ assert payload["prompt_token_ids"] == processed
+ assert chunk["engine_data"]["result"] == {NG_COMMIT_COORDS_FIELD: COORDS}
+
+
+def _cutting_host():
+ """A host whose checkpoint cuts stage their rows in memory."""
+ sink = _MemorySink()
+ host = _host(_FakeLLM())
+ host._capture_sink = sink
+ host.token_capture = RolloutTokenCapture(
+ sink=sink, weight_version_fn=lambda: 0, adapter=VLLMCaptureAdapter()
+ )
+ _cuts(host)
+ return host, sink
+
+
+def _cut(host, inventory: GenerationCutInventory) -> list[dict]:
+ """The frontend's cut of ``inventory``, sent to the engine."""
+ return _collect(
+ vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0),
+ {
+ "token_ids": [],
+ "annotations": [GENERATION_CUT_ANNOTATION],
+ "extra_args": {
+ INVENTORY_KEY: inventory.model_dump(mode="json"),
+ DEADLINE_KEY: time.time() + 30,
+ },
+ },
+ )
+
+
+def test_an_engine_answers_a_cut_with_the_calls_it_cut() -> None:
+ host, sink = _cutting_host()
+ inventory, _ = _active_call_inventory(host, ("held", [12, 13]))
+ # The frontend sends every engine the whole inventory: one call here,
+ # one that another engine holds.
+ inventory = GenerationCutInventory.build(
+ checkpoint_id=inventory.checkpoint_id,
+ server_name=inventory.server_name,
+ active_prefixes=[
+ *inventory.active_prefixes,
+ GenerationCutPrefix(
+ ticket_id="ticket-elsewhere",
+ rollout_id="r9",
+ attempt=0,
+ model_call_id="elsewhere",
+ admitted_at=1.0,
+ ),
+ ],
+ )
+
+ run_cut = host._prefix_recovery.run_cut
+ deadlines = []
+
+ async def record_deadline(inventory, *, deadline):
+ deadlines.append(deadline - time.monotonic())
+ return await run_cut(inventory, deadline=deadline)
+
+ host._prefix_recovery.run_cut = record_deadline
+
+ (chunk,) = _cut(host, inventory)
+
+ # The cut ran to the frontend's deadline, less the margin for the answer.
+ (remaining,) = deadlines
+ assert 30 - CUT_REPLY_MARGIN_S - 5 < remaining <= 30 - CUT_REPLY_MARGIN_S
+ (ack,) = chunk["engine_data"][CUT_PREFIXES_KEY]
+ assert (ack["model_call_id"], ack["disposition"]) == ("held", "durable_prefix")
+ assert ack["prefix_token_count"] == 2
+ # The cut's rows are in TQ before the engine answers.
+ assert ack["staging_keys"] == sink.generation_prefix_keys
+
+
+def test_a_cut_saves_a_media_calls_tensor_digests_in_its_first_row() -> None:
+ host, sink = _cutting_host()
+ _, _, media = _live_image_call(host)
+
+ _cut(host, _single_call_inventory())
+
+ ((_, row),) = sink.generation_prefix_records
+ assert row.extras[MEDIA_TENSOR_DIGESTS_FIELD] == media.tensor_digests
+
+
+@pytest.mark.parametrize(
+ "changed", [False, True], ids=["same_pixels", "changed_pixels"]
+)
+def test_a_restored_media_call_is_checked_against_its_cuts_tensor_digests(
+ changed: bool, monkeypatch
+) -> None:
+ saved_media = _captured_image_media()
+ image = [18, 18]
+ # The cut's first row: the prompt, one generated token and the media digests.
+ saved = SimpleNamespace(
+ model_call_id="c2",
+ token_ids_delta=[*image, 20],
+ token_mask_delta=[0.0, 0.0, 1.0],
+ prev_len=0,
+ extras={
+ MEDIA_SPANS_FIELD: [item.to_dict() for item in saved_media.items],
+ MEDIA_TENSOR_DIGESTS_FIELD: saved_media.tensor_digests,
+ },
+ )
+ admission = CaptureAdmission(
+ capture_key="rollout-1-a1",
+ model_call_id="c3",
+ mode="text",
+ generation_cut={
+ "source_capture_key": "rollout-1",
+ "source_model_call_id": "c2",
+ "staging_keys": ["active/checkpoint-1/rollout-1/c2/0"],
+ "prefix_token_count": 1,
+ "prefix_digest": "0" * 64,
+ "effective_output_limit": 8,
+ },
+ )
+ llm = _FakeLLM([_output([*image, 20], _completion([21], "stop"))])
+ host = _capturing_host(llm)
+ host._capture_media = host._hash_captured_media = True
+ monkeypatch.setattr(
+ host._prefix_recovery,
+ "resolve_restore",
+ lambda admission, prefix_token_ids: (admission, saved),
+ )
+ # What vLLM's processor made of the image the re-sent call carries.
+ tensors = dict(saved_media.tensors)
+ if changed:
+ tensors["imgs"] = tensors["imgs"] + 1
+ monkeypatch.setattr(
+ "nemo_rl.models.generation.vllm.vllm_worker_async.capture_processed_media",
+ lambda *args, **kwargs: CapturedMedia(saved_media.items, tensors),
+ )
+
+ (chunk,) = _collect(
+ vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0),
+ _captured_call(
+ image, admission, multimodal=_multimodal([PlaceholderRange(0, 2)])
+ ),
+ )
+
+ if changed:
+ # Refused before the call begins or generates.
+ refusal = chunk["engine_data"]["refusal"]
+ assert refusal["status"] == 400
+ assert "(retained_media_changed)" in refusal["message"]
+ assert (llm.calls, host.token_capture.begun) == ([], [])
+ return
+ # The engine resumes from the saved rows; the final row stages the pixels.
+ ((engine_input, _, _, _),) = llm.calls
+ assert engine_input["prompt_token_ids"] == [*image, 20]
+ _assert_stages_captured_tensors(host.token_capture.attachments[0], tensors)
+
+
+def test_a_captured_call_records_what_a_cut_needs() -> None:
+ """A checkpoint cuts only a call whose output budget and progress it knows."""
+ final = _output([1, 3, 4], _completion([10, 11], "stop"))
+ llm = _FakeLLM([final])
+ host = _capturing_host(llm)
+ host._prefix_recovery = MagicMock(wraps=NoPrefixRecovery())
+
+ _collect(
+ vllm_adapter.DynamoVllmAdapter(host, leader_worker_idx=0),
+ _captured_call([1, 3, 4], TEXT),
+ )
+
+ ((_, params, _, _),) = llm.calls
+ ((chat, recorded_params),) = [
+ call.args for call in host._prefix_recovery.on_sampling_params.call_args_list
+ ]
+ assert recorded_params is params
+ assert [call.args for call in host._prefix_recovery.observe.call_args_list] == [
+ (chat, final)
+ ]
diff --git a/tests/unit/models/generation/test_dynamo_vllm_decoupling.py b/tests/unit/models/generation/test_dynamo_vllm_decoupling.py
index 536f9e404f6..fff9809fc72 100644
--- a/tests/unit/models/generation/test_dynamo_vllm_decoupling.py
+++ b/tests/unit/models/generation/test_dynamo_vllm_decoupling.py
@@ -14,11 +14,12 @@
"""Dynamo stays out of NeMo RL's vLLM engines.
The engines run NeMo RL's own vLLM and use Dynamo only through its
-engine-agnostic backend SDK, so a vLLM bump can break Dynamo serving only in
-NeMo RL's own adapter (``dynamo/vllm_adapter.py``). These guards fail if
-NeMo RL starts importing Dynamo's vLLM integration (or any other Dynamo
-module), if the SDK itself starts importing vLLM, or if the Dynamo side shared
-by every backend starts importing one.
+engine-agnostic backend SDK, and the frontend serves chat with NeMo RL's own
+chat engine, so a vLLM bump can break Dynamo serving only in NeMo RL's own
+``dynamo/vllm/`` package. These guards fail if NeMo RL starts importing
+Dynamo's vLLM integration (or any other Dynamo module), if the Dynamo modules
+it imports start importing vLLM, or if the Dynamo side shared by every backend
+starts importing one.
"""
import ast
@@ -38,16 +39,30 @@
"dynamo.common.backend",
"dynamo.common.backend.logprobs",
"dynamo.common.backend.publisher",
+ # The frontend's entrypoint and chat-engine hook; the frontend loads Dynamo's
+ # own vLLM chat processor only for --dyn-chat-processor, which NeMo RL rejects.
+ "dynamo.frontend.main",
+ "dynamo.llm",
"dynamo.llm.exceptions",
}
-
-# The Dynamo side every generation backend builds on.
-BACKEND_NEUTRAL_MODULES = (
- "models/generation/dynamo/engine_adapter.py",
- "models/generation/rollout_result.py",
-)
GENERATION_BACKENDS = {"vllm", "sglang", "trtllm", "megatron"}
+DYNAMO_ROOT = NEMO_RL_ROOT / "models/generation/dynamo"
+
+
+def _backend_neutral_modules() -> list[str]:
+ """The Dynamo side every generation backend builds on: all of ``dynamo/``
+ outside its per-backend packages, and the wire types it shares."""
+ paths = [
+ path
+ for path in DYNAMO_ROOT.rglob("*.py")
+ if path.relative_to(DYNAMO_ROOT).parts[0] not in GENERATION_BACKENDS
+ ]
+ paths += [
+ NEMO_RL_ROOT / "models/generation/rollout_result.py",
+ NEMO_RL_ROOT / "models/generation/gym_call.py",
+ ]
+ return sorted(str(path.relative_to(NEMO_RL_ROOT)) for path in paths)
def _imports(path: Path) -> set[str]:
@@ -90,20 +105,22 @@ def test_nemo_rl_imports_only_dynamos_backend_sdk() -> None:
def test_the_shared_dynamo_side_imports_no_generation_backend() -> None:
+ modules = _backend_neutral_modules()
+ assert "models/generation/dynamo/engine_adapter.py" in modules
backends = {
path: sorted(
module
for module in _imports(NEMO_RL_ROOT / path)
if GENERATION_BACKENDS & set(module.split("."))
)
- for path in BACKEND_NEUTRAL_MODULES
+ for path in modules
}
- assert backends == {path: [] for path in BACKEND_NEUTRAL_MODULES}
+ assert backends == {path: [] for path in modules}
@pytest.mark.vllm
-def test_dynamos_backend_sdk_imports_no_vllm() -> None:
+def test_the_dynamo_modules_nemo_rl_imports_import_no_vllm() -> None:
pytest.importorskip("dynamo._core")
# A fresh interpreter, so modules other tests imported do not count.
code = (
diff --git a/tests/unit/models/generation/test_gym_call.py b/tests/unit/models/generation/test_gym_call.py
new file mode 100644
index 00000000000..ec31a63b82d
--- /dev/null
+++ b/tests/unit/models/generation/test_gym_call.py
@@ -0,0 +1,137 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""NeMo-Gym's Dynamo dialect packs and lifts the fields NeMo RL's call models define."""
+
+from types import SimpleNamespace
+
+import pytest
+
+pytestmark = pytest.mark.nemo_gym
+pytest.importorskip("nemo_gym.token_id_capture.external_capture")
+
+from nemo_gym.openai_utils import ( # noqa: E402
+ TOKEN_METADATA_FIELDS,
+ NeMoGymAsyncOpenAI,
+)
+from nemo_gym.token_id_capture import ( # noqa: E402
+ NG_CAPTURE_FIELD,
+ NG_COMMIT_COORDS_FIELD,
+)
+from nemo_gym.token_id_capture.external_capture import ( # noqa: E402
+ DYNAMO_GYM_EXTRAS_ANNOTATION_PREFIX,
+ add_dynamo_gym_extras,
+ pop_dynamo_call_result,
+)
+from responses_api_models.vllm_model.app import VLLMModel # noqa: E402
+from responses_api_models.vllm_model.tests import ( # noqa: E402
+ test_framework_context as gym_harness,
+)
+
+from nemo_rl.models.generation.gym_call import ( # noqa: E402
+ GYM_EXTRAS_ANNOTATION_PREFIX,
+ GYM_RESULT_NVEXT_KEY,
+ GymCallExtras,
+ GymCallResult,
+)
+
+
+def test_gyms_extras_are_the_ones_nemo_rl_reads() -> None:
+ assert DYNAMO_GYM_EXTRAS_ANNOTATION_PREFIX == GYM_EXTRAS_ANNOTATION_PREFIX
+ # Gym's own request preparation: the last echoed turn becomes the prefix.
+ body = {
+ "messages": [
+ {"role": "user", "content": "hi"},
+ {
+ "role": "assistant",
+ "content": "a",
+ "prompt_token_ids": [1, 2],
+ "generation_token_ids": [3],
+ "generation_log_probs": [-0.5],
+ },
+ ],
+ "nvext": {"annotations": ["other"]},
+ }
+ VLLMModel._prepare_dynamo_request(
+ SimpleNamespace(_external_capture_handler=None), body
+ )
+ # The capture handler's admission and the measure flag ride in the same object.
+ add_dynamo_gym_extras(
+ body, {NG_CAPTURE_FIELD: {"capture_key": "r"}, "ng_measure": True}
+ )
+
+ # Unknown names are refused, so a field Gym renames fails here.
+ extras = GymCallExtras.from_annotations(body["nvext"]["annotations"])
+ assert extras == GymCallExtras(
+ ng_capture={"capture_key": "r"},
+ required_prefix_token_ids=[1, 2, 3],
+ ng_measure=True,
+ )
+ assert body["nvext"]["annotations"][0] == "other"
+ assert "prompt_token_ids" not in body["messages"][1]
+
+
+def test_gym_lifts_the_call_result_nemo_rl_returns() -> None:
+ result = GymCallResult(
+ prompt_token_ids=[1, 2],
+ generation_token_ids=[3],
+ generation_log_probs=[-0.5],
+ routed_experts="nrlre1:int16:3x1x1:AAABAAIA",
+ ng_commit_coords={"staging_key": "r/c1"},
+ ).model_dump(exclude_none=True)
+
+ assert pop_dynamo_call_result({"nvext": {GYM_RESULT_NVEXT_KEY: result}}) == result
+ # The fields Gym copies onto the message, and its capture acknowledgement.
+ assert TOKEN_METADATA_FIELDS | {NG_COMMIT_COORDS_FIELD} <= result.keys()
+
+
+def test_gyms_measure_call_is_the_one_nemo_rl_answers(monkeypatch, tmp_path) -> None:
+ # Gym's own /context/measure, in its Dynamo dialect.
+ harness = gym_harness.make_capture_harness(monkeypatch, tmp_path, dynamo=True)
+ first = gym_harness.assert_clean(harness.post("attempt", gym_harness.HISTORY))
+ seen = []
+
+ async def frontend(client, *, extra_headers=None, **body):
+ # Unknown names are refused, so a measure flag Gym renames fails here.
+ seen.append(GymCallExtras.from_annotations(body["nvext"]["annotations"]))
+ result = GymCallResult(prompt_token_count=19, ng_context_measured=True)
+ return {
+ "id": "chatcmpl-measure",
+ "object": "chat.completion",
+ "created": 0,
+ "model": "m",
+ "choices": [
+ {
+ "index": 0,
+ "finish_reason": "stop",
+ "message": {"role": "assistant", "content": ""},
+ }
+ ],
+ "nvext": {GYM_RESULT_NVEXT_KEY: result.model_dump(exclude_none=True)},
+ }
+
+ monkeypatch.setattr(NeMoGymAsyncOpenAI, "create_chat_completion", frontend)
+ response = harness.client.post(
+ "/ng-rollout/attempt/training-token-capture/context/measure",
+ json={
+ "input": gym_harness.HISTORY
+ + first["output"]
+ + [{"role": "user", "content": "next"}]
+ },
+ )
+
+ # Gym read the result back by NeMo RL's names; a renamed one answers 502.
+ assert response.status_code == 200, response.text
+ assert response.json() == {"prompt_token_count": 19}
+ (extras,) = seen
+ assert extras.ng_measure is True
diff --git a/tests/unit/models/generation/test_gym_chat_path.py b/tests/unit/models/generation/test_gym_chat_path.py
new file mode 100644
index 00000000000..bd90d96d0b3
--- /dev/null
+++ b/tests/unit/models/generation/test_gym_chat_path.py
@@ -0,0 +1,1291 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""NeMo-Gym's chat calls take the same steps on both of NeMo RL's serving paths.
+
+Each case sends one call to the native vLLM server (its real routes, over ASGI)
+and through Dynamo (NeMo RL's chat engine, a spawned render pool, and the
+engine adapter), with the same kit (a checkpoint and the chat settings both
+paths serve), the same scripted engine and Gym's real capture core. Both must
+run the engine on the same prompt, media and SamplingParams, stage the same
+capture record, and answer Gym with the same message and token fields.
+
+Not covered here: staging an image's pixels, which needs the Omni processor
+and its gated checkpoint.
+"""
+
+import asyncio
+import base64
+import io
+import json
+import shutil
+from dataclasses import dataclass, field
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Callable, Optional
+
+import numpy as np
+import pytest
+import torch
+
+pytestmark = [pytest.mark.vllm, pytest.mark.nemo_gym]
+pytest.importorskip("dynamo._core")
+pytest.importorskip("vllm")
+pytest.importorskip("nemo_gym.token_id_capture.staging")
+
+import httpx # noqa: E402
+from dynamo.llm.exceptions import HttpError # noqa: E402
+from fastapi import FastAPI # noqa: E402
+from nemo_gym._checkpoint.generation_cut import ( # noqa: E402
+ GenerationCutInventory,
+ GenerationCutPrefix,
+)
+from nemo_gym.token_id_capture.adapters.vllm import VLLMCaptureAdapter # noqa: E402
+from nemo_gym.token_id_capture.external_capture import ( # noqa: E402
+ split_dynamo_capture_refusal,
+)
+from nemo_gym.token_id_capture.protocols import CAPTURE_REFUSAL_HEADER # noqa: E402
+from nemo_gym.token_id_capture.replay import ( # noqa: E402
+ replay_context,
+ summarize_replay,
+)
+from nemo_gym.token_id_capture.staging.capture import ( # noqa: E402
+ RolloutTokenCapture,
+)
+from nemo_gym.token_id_capture.staging.records import ( # noqa: E402
+ CaptureAdmission,
+ StageResult,
+)
+from PIL import Image # noqa: E402
+from transformers import AutoConfig, AutoProcessor, AutoTokenizer # noqa: E402
+from vllm.config import CacheConfig, LoadConfig, VllmConfig # noqa: E402
+from vllm.engine.arg_utils import AsyncEngineArgs # noqa: E402
+from vllm.exceptions import QueueOverflowError, VLLMValidationError # noqa: E402
+from vllm.logprobs import Logprob # noqa: E402
+from vllm.outputs import CompletionOutput, RequestOutput # noqa: E402
+from vllm.renderers.registry import renderer_from_config # noqa: E402
+from vllm.v1.engine.input_processor import InputProcessor # noqa: E402
+
+from nemo_rl.data_plane.tq_token_sink import ( # noqa: E402
+ ChainPrefixCache,
+ StagedPrefix,
+)
+from nemo_rl.models.generation.dynamo.vllm import chat_engine # noqa: E402
+from nemo_rl.models.generation.dynamo.vllm.adapter import ( # noqa: E402
+ DynamoVllmAdapter,
+ chat_runtime_data,
+)
+from nemo_rl.models.generation.dynamo.vllm.render_pool import RenderPool # noqa: E402
+from nemo_rl.models.generation.gym_call import ( # noqa: E402
+ GYM_EXTRAS_ANNOTATION_PREFIX,
+)
+from nemo_rl.models.generation.prefix_recovery.restore import ( # noqa: E402
+ _build_restored_prefix_terminal_output,
+ restored_terminal,
+)
+from nemo_rl.models.generation.vllm import chat_server # noqa: E402
+from nemo_rl.models.generation.vllm.utils import take_reasoning_ended # noqa: E402
+from nemo_rl.models.generation.vllm.vllm_worker_async import ( # noqa: E402
+ VllmAsyncGenerationWorkerImpl,
+)
+from tests.unit.models.generation.test_vllm_token_capture_hosting import ( # noqa: E402
+ _cuts,
+ _MemoryPrefixSource,
+ _MemorySink,
+)
+
+MAX_MODEL_LEN = 512
+GENERATION = {
+ "temperature": 0.7,
+ "top_p": 0.9,
+ "top_k": None,
+ "val_temperature": 0.0,
+ "val_top_p": 1.0,
+ "val_top_k": None,
+ "stop_token_ids": None,
+ "max_new_tokens": 64,
+ "vllm_cfg": {
+ "max_model_len": MAX_MODEL_LEN,
+ "http_server_serving_chat_kwargs": {
+ "tool_parser": "hermes",
+ "reasoning_parser": "qwen3",
+ "enable_auto_tools": True,
+ # Rendered beneath the request's own on both paths.
+ "default_chat_template_kwargs": {"enable_thinking": False},
+ },
+ },
+}
+TOOLS = [
+ {
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "The weather in a city.",
+ "parameters": {
+ "type": "object",
+ "properties": {"city": {"type": "string"}},
+ "required": ["city"],
+ },
+ },
+ }
+]
+# A tool whose schema has a null default, as pydantic writes an Optional field.
+NULL_DEFAULT_TOOLS = [
+ {
+ "type": "function",
+ "function": {
+ **TOOLS[0]["function"],
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "city": {"type": "string"},
+ "unit": {
+ "anyOf": [{"type": "string"}, {"type": "null"}],
+ "default": None,
+ },
+ },
+ "required": ["city"],
+ },
+ },
+ }
+]
+# Turn 1 as Gym sends it back: the template drops its reasoning when it
+# re-renders the history, so the model's own tokens differ from the render.
+HISTORY = [
+ {"role": "user", "content": "Weather in Paris?"},
+ {"role": "assistant", "content": "It is sunny.", "reasoning_content": "easy"},
+]
+OBSERVATION = {"role": "user", "content": "And in Lyon?"}
+REPLY = (
+ "\nchecking\n\n\n\n"
+ '{"name": "get_weather", "arguments": {"city": "Lyon"}}\n'
+)
+TURN_1_REPLY = "\neasy\n\n\nIt is sunny."
+# Reasoning the template drops from history, longer than the context.
+LONG_TURN_1_REPLY = "\n" + "hmm " * MAX_MODEL_LEN + "\n\n\nIt is sunny."
+# Turn 1 as a tool call, which Gym's source history recorded with empty
+# content and the harness now sends with none.
+TOOL_TURN = {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call-0",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'},
+ }
+ ],
+}
+TOOL_HISTORY = [HISTORY[0], {**TOOL_TURN, "content": ""}]
+TOOL_RESULT = {"role": "tool", "tool_call_id": "call-0", "content": "sunny"}
+
+
+def _image_url() -> str:
+ image = Image.new("RGB", (56, 56))
+ image.putdata([(4 * x, 4 * y, 128) for y in range(56) for x in range(56)])
+ png = io.BytesIO()
+ image.save(png, format="PNG")
+ return "data:image/png;base64," + base64.b64encode(png.getvalue()).decode()
+
+
+IMAGE_TURN = {
+ "role": "user",
+ "content": [
+ {"type": "image_url", "image_url": {"url": _image_url()}},
+ {"type": "text", "text": "And here?"},
+ ],
+}
+NANO_V3_PLUGIN = str(
+ Path(__file__).parents[4]
+ / "nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py"
+)
+
+
+@dataclass
+class _Kit:
+ """The checkpoint, and the vLLM objects both paths share."""
+
+ path: str
+ generation: dict[str, Any]
+ capture_media: bool
+ tokenizer: Any
+ engine_args: Any
+ model_config: Any
+ vllm_config: Any
+ renderer: Any
+ pool: RenderPool
+
+ def ids(self, text: str) -> list[int]:
+ return self.tokenizer.encode(text, add_special_tokens=False)
+
+ def turn_1_tokens(self, reply: str = TURN_1_REPLY) -> list[int]:
+ """Turn 1's prompt and the model's own reply, as capture staged them."""
+ prompt = self.tokenizer.apply_chat_template(
+ HISTORY[:1], tools=TOOLS, add_generation_prompt=True, tokenize=False
+ )
+ return self.ids(prompt + reply + "<|im_end|>")
+
+
+class _Engine:
+ """vLLM's engine client: answers every call with REPLY and records its runs.
+
+ ``refuse`` is what the engine refuses every call with, and where: its
+ queue check ("admission"), its input check ("generate"), or mid-generation
+ ("running"), where a cancelled native handler gets its CancelledError.
+ """
+
+ def __init__(self, kit: _Kit, refuse: Optional[tuple[str, Any]] = None) -> None:
+ self.model_config = kit.model_config
+ self.renderer = kit.renderer
+ self.vllm_config = kit.vllm_config
+ self.input_processor = InputProcessor(kit.vllm_config)
+ self.io_processor = None
+ self.errored = False
+ self._kit = kit
+ self._refuse = refuse
+ self.runs: list[tuple[list[int], Any, Any]] = []
+ # The reasoning_ended each run was handed.
+ self.reasoning_ended: list[Optional[bool]] = []
+ # How many of REPLY's tokens the prompt already ends with: a call
+ # resumed from a checkpoint cut generates only the rest.
+ self.resumed = 0
+
+ async def generate(self, prompt, sampling_params, request_id, **kwargs):
+ if self._refuse is not None and self._refuse[0] == "generate":
+ raise self._refuse[1]
+ # As the worker's engine client does before it submits the call.
+ take_reasoning_ended(sampling_params, kwargs)
+ prompt_token_ids = list(prompt["prompt_token_ids"])
+ self.runs.append((prompt_token_ids, sampling_params, _media(prompt)))
+ self.reasoning_ended.append(kwargs.get("reasoning_ended"))
+ if self._refuse is not None and self._refuse[0] == "running":
+ raise self._refuse[1]
+ terminal = restored_terminal()
+ if terminal is not None:
+ # As the worker's engine client answers a call whose saved prefix
+ # had already finished it: without decoding.
+ yield _build_restored_prefix_terminal_output(request_id, terminal)
+ return
+ reply_ids = self._kit.ids(REPLY)[self.resumed :]
+ logprobs = None
+ if sampling_params.logprobs is not None:
+ decode = self._kit.tokenizer.decode
+ # The sampled token, and the runner-up a top_logprobs > 0 call sees.
+ logprobs = [
+ {
+ token_id: Logprob(-0.5, 1, decode([token_id])),
+ token_id + 1: Logprob(-2.0, 2, decode([token_id + 1])),
+ }
+ for token_id in reply_ids
+ ]
+ routes = None
+ if self._kit.generation.get("vllm_kwargs", {}).get(
+ "enable_return_routed_experts"
+ ):
+ # vLLM routes every token but the last sampled one: [tokens, layers, k].
+ tokens = len(prompt_token_ids) + len(reply_ids) - 1
+ routes = np.arange(tokens * 4, dtype=np.int16).reshape(tokens, 2, 2)
+ output = RequestOutput(
+ request_id=request_id,
+ prompt=None,
+ prompt_token_ids=prompt_token_ids,
+ prompt_logprobs=None,
+ outputs=[
+ CompletionOutput(
+ index=0,
+ text=self._kit.tokenizer.decode(reply_ids)
+ if self.resumed
+ else REPLY,
+ token_ids=reply_ids,
+ cumulative_logprob=None,
+ logprobs=logprobs,
+ finish_reason="stop",
+ routed_experts=None
+ if routes is None
+ else routes[len(prompt_token_ids) :],
+ )
+ ],
+ finished=True,
+ num_cached_tokens=0,
+ )
+ if routes is not None:
+ output.prompt_routed_experts = routes[: len(prompt_token_ids)]
+ yield output
+
+ def check_admission(self, n, request_id=None) -> None:
+ if self._refuse is not None and self._refuse[0] == "admission":
+ raise self._refuse[1]
+
+ async def is_tracing_enabled(self) -> bool:
+ return False
+
+ async def abort(self, request_id) -> None:
+ pass
+
+
+def _media(prompt) -> Any:
+ """The processed media an engine prompt runs: its token ranges and tensors."""
+ if "mm_kwargs" not in prompt:
+ return None
+ return (
+ {
+ modality: [(item.offset, item.length) for item in ranges]
+ for modality, ranges in prompt["mm_placeholders"].items()
+ },
+ {
+ modality: [{key: elem.data for key, elem in item.items()} for item in items]
+ for modality, items in prompt["mm_kwargs"].items()
+ },
+ )
+
+
+def _assert_same_media(dynamo, native) -> None:
+ if native is None:
+ assert dynamo is None
+ return
+ assert dynamo[0] == native[0]
+ assert dynamo[1].keys() == native[1].keys()
+ for modality, items in native[1].items():
+ assert len(dynamo[1][modality]) == len(items)
+ for dynamo_item, item in zip(dynamo[1][modality], items):
+ assert dynamo_item.keys() == item.keys()
+ for key, tensor in item.items():
+ assert torch.equal(dynamo_item[key], tensor), (modality, key)
+
+
+class _Sink:
+ def __init__(self) -> None:
+ self.staged: list[tuple[Any, Any]] = []
+
+ def stage(self, record, *, attachments=None) -> StageResult:
+ self.staged.append((record, attachments))
+ return StageResult(ok=True, staging_key=record.staging_key)
+
+
+class _Staged:
+ """TQ's staged calls: each call's token delta, by staging key."""
+
+ def __init__(self, deltas: dict[str, list[int]]) -> None:
+ self.deltas = deltas
+
+ def fetch_prefix_chains(self, staging_keys: list[str]) -> StagedPrefix:
+ return StagedPrefix(
+ expanded=[token for key in staging_keys for token in self.deltas[key]]
+ )
+
+
+@dataclass
+class _Outcome:
+ """What one path did with a call."""
+
+ # None when the client went away: nobody is answered.
+ status: Optional[int]
+ reply: dict[str, Any]
+ runs: list[tuple[list[int], Any, Any]]
+ staged: list[tuple[Any, Any]]
+ failed: list[str] = field(default_factory=list)
+ # The capture call Gym is told was refused before it was staged, if any.
+ acknowledged: Optional[str] = None
+ # What the engine was told of each run's reasoning.
+ reasoning_ended: list[Optional[bool]] = field(default_factory=list)
+
+
+def _host(
+ kit: _Kit,
+ deltas: dict[str, list[int]],
+ top_logprobs: int,
+ refuse: Optional[tuple[str, Any]] = None,
+):
+ host = object.__new__(VllmAsyncGenerationWorkerImpl)
+ host._init_token_capture_state()
+ host.cfg = kit.generation
+ host.llm = host._http_engine_client = _Engine(kit, refuse)
+ host.llm_async_engine_args = kit.engine_args
+ host.model_name = "model"
+ host.routed_experts_dtype = torch.int16
+ host._capture_media = kit.capture_media
+ host._capture_top_logprobs = top_logprobs
+ host._staging_source = None
+ host._rollout_weight_version = 0
+ host._chain_prefix = ChainPrefixCache()
+ host._chain_prefix.install(_Staged(deltas))
+ sink = _Sink()
+ capture = RolloutTokenCapture(
+ sink=sink, weight_version_fn=lambda: 0, adapter=VLLMCaptureAdapter()
+ )
+ failed: list[str] = []
+ fail_call = capture.fail_call
+
+ def record_failure(call, *, reason):
+ failed.append(reason)
+ return fail_call(call, reason=reason)
+
+ capture.fail_call = record_failure
+ host.token_capture = capture
+ return host, sink, failed
+
+
+async def _native(kit, host, sink, failed, body, *, measure: bool) -> _Outcome:
+ server = chat_server.build_chat_server(
+ host,
+ engine_client=host._http_engine_client,
+ model_config=kit.model_config,
+ cfg=kit.generation,
+ )
+ app = FastAPI()
+ chat_server.register_routes(
+ app, server, host=host, generation_config=kit.generation
+ )
+ async with httpx.AsyncClient(
+ # A server error is answered with a 500, as uvicorn does.
+ transport=httpx.ASGITransport(app=app, raise_app_exceptions=False),
+ base_url="http://native",
+ ) as client:
+ try:
+ response = await client.post(
+ "/tokenize" if measure else "/v1/chat/completions", json=body
+ )
+ except asyncio.CancelledError:
+ # The handler was cancelled, as on a client disconnect.
+ return _Outcome(None, {}, host.llm.runs, sink.staged, failed)
+ is_json = response.headers["content-type"] == "application/json"
+ return _Outcome(
+ response.status_code,
+ response.json() if is_json else {"error": response.text},
+ host.llm.runs,
+ sink.staged,
+ failed,
+ response.headers.get(CAPTURE_REFUSAL_HEADER),
+ host.llm.reasoning_ended,
+ )
+
+
+class _Router:
+ """Dynamo's router to one engine: the request crosses the wire as JSON."""
+
+ def __init__(self, adapter: DynamoVllmAdapter, *, stopped: bool) -> None:
+ self._adapter = adapter
+ self._stopped = stopped
+
+ async def generate(self, request, context=None):
+ wire = json.loads(json.dumps(request))
+ context = _Context(self._stopped)
+
+ async def stream():
+ try:
+ async for chunk in self._adapter.generate(wire, context):
+ yield SimpleItem(chunk)
+ except Exception as error:
+ # Dynamo raises an engine's error from the stream, as a ValueError.
+ raise ValueError(str(error)) from error
+
+ return stream()
+
+
+class _Context:
+ def __init__(self, stopped: bool) -> None:
+ self._stopped = stopped
+
+ def id(self) -> str:
+ return "chatcmpl-0"
+
+ def is_stopped(self) -> bool:
+ return self._stopped
+
+
+class SimpleItem:
+ def __init__(self, data) -> None:
+ self._data = data
+
+ def data(self):
+ return self._data
+
+
+async def _dynamo(
+ kit, host, sink, failed, body, extras, *, stopped: bool = False
+) -> _Outcome:
+ request = {k: v for k, v in body.items() if k not in extras}
+ # Dynamo's request names the template kwargs chat_template_args.
+ if "chat_template_kwargs" in request:
+ request["chat_template_args"] = request.pop("chat_template_kwargs")
+ request["nvext"] = {
+ "annotations": [GYM_EXTRAS_ANNOTATION_PREFIX + json.dumps(extras)]
+ }
+ capture = chat_engine._CaptureSource()
+ capture._prefix_cache = host._chain_prefix
+ engine = chat_engine.NemoRLChatEngine(
+ pool=kit.pool,
+ routed_engine=_Router(
+ DynamoVllmAdapter(host, leader_worker_idx=0), stopped=stopped
+ ),
+ spec=json.loads(json.dumps(chat_runtime_data(host)))[
+ chat_engine.RUNTIME_DATA_KEY
+ ],
+ capture=capture,
+ )
+ try:
+ (chunk,) = [chunk async for chunk in engine.generate(request, None)]
+ except HttpError as error:
+ message, acknowledged = split_dynamo_capture_refusal(error.message)
+ return _Outcome(
+ # The frontend's answer to a client that went away reaches nobody.
+ None if stopped else error.code,
+ {"error": message},
+ host.llm.runs,
+ sink.staged,
+ failed,
+ acknowledged,
+ )
+ return _Outcome(
+ 200,
+ chunk,
+ host.llm.runs,
+ sink.staged,
+ failed,
+ reasoning_ended=host.llm.reasoning_ended,
+ )
+
+
+def _generation(*, vllm_cfg=None, **chat_settings) -> dict[str, Any]:
+ """GENERATION with the native chat settings a recipe changes."""
+ base = GENERATION["vllm_cfg"]
+ return {
+ **GENERATION,
+ "vllm_cfg": {
+ **base,
+ **(vllm_cfg or {}),
+ "http_server_serving_chat_kwargs": {
+ **base["http_server_serving_chat_kwargs"],
+ **chat_settings,
+ },
+ },
+ }
+
+
+def _turn_end_not_eos(path: str, tmp_path_factory) -> str:
+ """The checkpoint, shaped like gpt-oss: its turns end on a generation-config
+ EOS (<|im_end|>) that is not the tokenizer's (<|endoftext|>)."""
+ copy = tmp_path_factory.mktemp("turn_end_not_eos")
+ shutil.copytree(path, copy, dirs_exist_ok=True)
+ tokenizer_config = json.loads((copy / "tokenizer_config.json").read_text())
+ tokenizer = AutoTokenizer.from_pretrained(path)
+ eos = tokenizer.convert_tokens_to_ids(["<|im_end|>", "<|endoftext|>"])
+ tokenizer_config["eos_token"] = "<|endoftext|>"
+ (copy / "tokenizer_config.json").write_text(json.dumps(tokenizer_config))
+ (copy / "generation_config.json").write_text(json.dumps({"eos_token_id": eos}))
+ return str(copy)
+
+
+def _vlm(path: str, tmp_path_factory) -> str:
+ """Qwen2.5-VL's config and processor: rendering never loads weights."""
+ del path
+ checkpoint = tmp_path_factory.mktemp("qwen2_5_vl")
+ AutoConfig.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct").save_pretrained(
+ checkpoint
+ )
+ AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct").save_pretrained(
+ checkpoint
+ )
+ return str(checkpoint)
+
+
+@dataclass(frozen=True)
+class _KitSpec:
+ generation: dict[str, Any]
+ # The checkpoint, from the tiny Qwen3's path.
+ checkpoint: Callable[..., str] = lambda path, _: path
+ capture_media: bool = False
+
+
+KITS = {
+ "qwen3": _KitSpec(GENERATION),
+ "turn_end_not_eos": _KitSpec(GENERATION, _turn_end_not_eos),
+ # The template gets structured content parts instead of strings.
+ "structured_content": _KitSpec(_generation(chat_template_content_format="openai")),
+ # Nemotron-3-Nano's parser, a plugin file the native recipe loads.
+ "nano_v3": _KitSpec(
+ _generation(
+ vllm_cfg={"reasoning_parser_plugin": NANO_V3_PLUGIN},
+ reasoning_parser="nano_v3",
+ )
+ ),
+ # Qwen2.5's tokenizer has no thinking tokens for Qwen3's reasoning parser.
+ "vlm": _KitSpec(_generation(reasoning_parser=""), _vlm),
+ # A tool parser with no structural tag: a forced tool sets structured outputs.
+ "pythonic": _KitSpec(_generation(tool_parser="pythonic")),
+ # A reasoning parser that reads the prompt back to its last think token.
+ "deepseek_r1": _KitSpec(_generation(reasoning_parser="deepseek_r1")),
+ # Router replay: the engine returns each token's routed experts.
+ "router_replay": _KitSpec(
+ {**GENERATION, "vllm_kwargs": {"enable_return_routed_experts": True}}
+ ),
+}
+
+
+def _build_kit(kit_spec: _KitSpec, path: str) -> _Kit:
+ generation = kit_spec.generation
+ # No processor cache: each case's engine gets the media itself, not a handle
+ # to media an earlier case processed.
+ engine_args = AsyncEngineArgs(
+ model=path, max_model_len=MAX_MODEL_LEN, mm_processor_cache_gb=0
+ )
+ model_config = engine_args.create_model_config()
+ vllm_config = VllmConfig(
+ model_config=model_config,
+ load_config=LoadConfig(load_format="dummy"),
+ cache_config=CacheConfig(),
+ )
+ # The pool renders with what an engine publishes at registration.
+ spec = json.loads(
+ json.dumps(
+ chat_runtime_data(
+ SimpleNamespace(llm_async_engine_args=engine_args, cfg=generation)
+ )
+ )
+ )[chat_engine.RUNTIME_DATA_KEY]
+ pool = RenderPool(workers=1)
+ pool.build(spec, path)
+ return _Kit(
+ path=path,
+ generation=generation,
+ capture_media=kit_spec.capture_media,
+ tokenizer=AutoTokenizer.from_pretrained(path),
+ engine_args=engine_args,
+ model_config=model_config,
+ vllm_config=vllm_config,
+ renderer=renderer_from_config(vllm_config),
+ pool=pool,
+ )
+
+
+@pytest.fixture(scope="module")
+def kits(tiny_qwen3_model_path, tmp_path_factory):
+ """Each kit in KITS, built on first use."""
+ built: dict[str, _Kit] = {}
+
+ def kit(name: str) -> _Kit:
+ if name not in built:
+ spec = KITS[name]
+ built[name] = _build_kit(
+ spec, spec.checkpoint(tiny_qwen3_model_path, tmp_path_factory)
+ )
+ return built[name]
+
+ yield kit
+ for built_kit in built.values():
+ built_kit.pool.shutdown()
+
+
+def _admission(**fields) -> dict[str, Any]:
+ return CaptureAdmission(
+ capture_key="rollout-1", model_call_id="c2", **fields
+ ).model_dump(mode="json")
+
+
+def _staged_turn_1(kit: _Kit, reply: str = TURN_1_REPLY, **fields) -> dict[str, Any]:
+ """Turn 2's admission, chained to turn 1 staged under rollout-1/c1."""
+ return _admission(
+ parent_call_id="c1",
+ prev_len=len(kit.turn_1_tokens(reply)),
+ parent_chain_hash="1" * 64,
+ staging_chain=["rollout-1/c1"],
+ **fields,
+ )
+
+
+def _candidate(
+ kit: _Kit,
+ messages: list[dict],
+ source: list[dict] = HISTORY,
+ *,
+ empty_tool_content: bool = False,
+) -> dict[str, Any]:
+ """Gym's candidate: turn 1's source, and this request's, for RL to check the cut."""
+ return _staged_turn_1(
+ kit,
+ mode="candidate",
+ requires_empty_tool_content_equivalence=empty_tool_content,
+ candidate_replay=summarize_replay(
+ replay_context(source, render_digest="b" * 64)
+ ),
+ request_replay=replay_context(messages, render_digest="b" * 64),
+ )
+
+
+@dataclass(frozen=True)
+class _Case:
+ """One Gym call, and what both paths must do with it."""
+
+ id: str
+ messages: list[dict] = field(default_factory=lambda: [*HISTORY, OBSERVATION])
+ extras: Callable[[_Kit], dict[str, Any]] = lambda kit: {}
+ fields: dict[str, Any] = field(default_factory=dict)
+ # The sampler top-k capture stages, for score centering.
+ top_logprobs: int = 0
+ # Request fields the call leaves out.
+ omit: tuple[str, ...] = ()
+ # 200, or the status both paths answer a call that did not finish with.
+ status: int = 200
+ # The mode capture stages the call in; None when the call is not captured.
+ mode: Optional[str] = None
+ kit: str = "qwen3"
+ # What the engine refuses the call with, and where (see _Engine).
+ refuse: Optional[tuple[str, Any]] = None
+ # The client goes away while the engine generates: nobody is answered.
+ disconnect: bool = False
+ # What an unfinished call's capture is failed with, on both paths.
+ failed: tuple[str, ...] = ()
+ # Whether Gym is told the capture call was refused before it was staged.
+ acknowledged: bool = False
+ # Turn 1's reply as the model wrote it and capture staged it.
+ turn_1_reply: str = TURN_1_REPLY
+ # What the engine is told of the call's reasoning; unchecked when None.
+ reasoning_ended: Optional[bool] = None
+
+
+def _off_policy(name: str, value: Any) -> _Case:
+ return _Case(f"off_policy_{name}", fields={name: value}, status=400)
+
+
+CASES = [
+ # A plain call: the render and the reply's reasoning and tool parsers.
+ _Case("plain"),
+ # No max_tokens: each path fills the budget the way vLLM's server does.
+ _Case("no_max_tokens", fields={"max_tokens": None}),
+ # Gym's echo of turn 1 without capture: the model's tokens replace the
+ # template's re-render of that turn (the turn-end cut).
+ _Case(
+ "inline_prefix",
+ extras=lambda kit: {"required_prefix_token_ids": kit.turn_1_tokens()},
+ ),
+ # The same, for a model whose turns end on a token that is not its
+ # tokenizer's EOS, as gpt-oss's <|call|>.
+ _Case(
+ "turn_end_not_eos",
+ extras=lambda kit: {"required_prefix_token_ids": kit.turn_1_tokens()},
+ kit="turn_end_not_eos",
+ ),
+ # Token capture: turn 1's staged tokens, read from TQ.
+ _Case(
+ "token_in",
+ extras=lambda kit: {"ng_capture": _staged_turn_1(kit, mode="token_in")},
+ mode="token_in",
+ ),
+ # Gym's candidate whose cut holds is served token-in...
+ _Case(
+ "candidate_cut_holds",
+ extras=lambda kit: {"ng_capture": _candidate(kit, [*HISTORY, OBSERVATION])},
+ mode="token_in",
+ ),
+ # ...and one whose conversion moved the cut roots a new text segment.
+ _Case(
+ "candidate_cut_fails",
+ messages=[*HISTORY, OBSERVATION, HISTORY[-1]],
+ extras=lambda kit: {"ng_capture": _candidate(kit, [*HISTORY, OBSERVATION])},
+ mode="text",
+ ),
+ # A candidate whose tool call had empty content and now has none: string
+ # content renders both alike, so the cut holds...
+ _Case(
+ "empty_tool_content",
+ messages=[HISTORY[0], TOOL_TURN, TOOL_RESULT],
+ extras=lambda kit: {
+ "ng_capture": _candidate(
+ kit,
+ [HISTORY[0], TOOL_TURN, TOOL_RESULT],
+ TOOL_HISTORY,
+ empty_tool_content=True,
+ )
+ },
+ mode="token_in",
+ ),
+ # ...and structured content cannot prove that, so it roots a new segment.
+ _Case(
+ "empty_tool_content_structured",
+ messages=[HISTORY[0], TOOL_TURN, TOOL_RESULT],
+ extras=lambda kit: {
+ "ng_capture": _candidate(
+ kit,
+ [HISTORY[0], TOOL_TURN, TOOL_RESULT],
+ TOOL_HISTORY,
+ empty_tool_content=True,
+ )
+ },
+ mode="text",
+ kit="structured_content",
+ ),
+ # Score centering stages the sampler's top-k with the call.
+ _Case(
+ "score_centering",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ fields={"top_logprobs": 2},
+ top_logprobs=2,
+ mode="text",
+ ),
+ # Capture with a processor that has no media capture is refused before it
+ # runs, with the native server's 400.
+ _Case(
+ "image_capture_unsupported",
+ messages=[IMAGE_TURN],
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ status=400,
+ kit="vlm",
+ ),
+ # An image after the echoed turn 1: processed once in the frontend, both
+ # engines run the same pixels, and the splice moves its tokens.
+ _Case(
+ "image_after_echoed_turn",
+ messages=[*HISTORY, IMAGE_TURN],
+ extras=lambda kit: {"required_prefix_token_ids": kit.turn_1_tokens()},
+ kit="vlm",
+ ),
+ # The client's nulls in a tool schema are rendered into the prompt.
+ _Case("tool_schema_null_default", fields={"tools": NULL_DEFAULT_TOOLS}),
+ # A named tool_choice, whose schema the tool parser sets as structured outputs.
+ _Case(
+ "named_tool_choice",
+ fields={
+ "tool_choice": {"type": "function", "function": {"name": "get_weather"}}
+ },
+ kit="pythonic",
+ ),
+ # Nemotron-3-Nano's reasoning parser plugin, with thinking off.
+ _Case(
+ "nano_v3_thinking_off",
+ fields={"chat_template_kwargs": {"enable_thinking": False}},
+ kit="nano_v3",
+ ),
+ # The sampling the trainer replays, set explicitly...
+ _Case(
+ "on_policy_sampling",
+ fields={
+ "top_k": -1,
+ "min_p": 0.0,
+ "repetition_penalty": 1.0,
+ "presence_penalty": 0.0,
+ "frequency_penalty": 0.0,
+ "logit_bias": {},
+ },
+ ),
+ # ...and any it does not is refused before anything runs.
+ _off_policy("temperature", 0.5),
+ _off_policy("top_p", 0.5),
+ _off_policy("top_k", 50),
+ _off_policy("min_p", 0.1),
+ _off_policy("repetition_penalty", 1.05),
+ _off_policy("presence_penalty", 0.5),
+ _off_policy("frequency_penalty", 0.5),
+ _off_policy("logit_bias", {"10": 5.0}),
+ # Token information without top_logprobs, unset or sent as null, is served as 0.
+ _Case("top_logprobs_unset", omit=("top_logprobs",)),
+ _Case("top_logprobs_null", fields={"top_logprobs": None}),
+ # A field vLLM refuses while the request is parsed, before any step runs.
+ _Case(
+ "field_error",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ fields={"logprobs": False, "top_logprobs": 5},
+ status=400,
+ acknowledged=True,
+ ),
+ # The model's own turn 1 is longer than the template's render of it, so the
+ # spliced prompt fills the context though the render fits.
+ _Case(
+ "spliced_prompt_overflow",
+ extras=lambda kit: {
+ "ng_capture": _staged_turn_1(kit, LONG_TURN_1_REPLY, mode="token_in")
+ },
+ status=400,
+ acknowledged=True,
+ turn_1_reply=LONG_TURN_1_REPLY,
+ ),
+ # A full queue: vLLM's 503, so Gym retries elsewhere; checked before capture.
+ _Case(
+ "queue_full",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ status=503,
+ refuse=("admission", QueueOverflowError()),
+ ),
+ # The engine's input check refuses a captured call after it began.
+ _Case(
+ "token_out_of_vocabulary",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ status=400,
+ refuse=(
+ "generate",
+ VLLMValidationError("Token id 999999 is out of vocabulary"),
+ ),
+ failed=("invalid_request",),
+ acknowledged=True,
+ ),
+ # The engine fails a captured call: a server error, and nothing staged.
+ _Case(
+ "engine_error",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ status=500,
+ refuse=("generate", RuntimeError("engine failed")),
+ failed=("engine_error",),
+ ),
+ # The client goes away mid-generation: the call ran, and is not staged.
+ _Case(
+ "client_disconnect",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ disconnect=True,
+ failed=("client_disconnect",),
+ ),
+ # Structured output after the model's own turn 1, whose reasoning the
+ # template drops: that turn's does not end this turn's reasoning.
+ _Case(
+ "reasoning_not_ended_by_an_earlier_turn",
+ extras=lambda kit: {"required_prefix_token_ids": kit.turn_1_tokens()},
+ fields={
+ "chat_template_kwargs": {"enable_thinking": True},
+ "response_format": {"type": "json_object"},
+ },
+ kit="deepseek_r1",
+ reasoning_ended=False,
+ ),
+ # Router replay stages each token's routed experts with the call.
+ _Case(
+ "router_replay",
+ extras=lambda kit: {"ng_capture": _admission(mode="text")},
+ mode="text",
+ kit="router_replay",
+ ),
+]
+
+
+@pytest.mark.parametrize("case", CASES, ids=[case.id for case in CASES])
+def test_a_gym_call_takes_the_same_steps_through_dynamo_as_natively(kits, case) -> None:
+ kit = kits(case.kit)
+ extras = case.extras(kit)
+ body = {
+ "model": kit.path,
+ "messages": case.messages,
+ "tools": TOOLS,
+ "temperature": 0.7,
+ "top_p": 0.9,
+ "max_tokens": 32,
+ # Gym's token information.
+ "logprobs": True,
+ "top_logprobs": 0,
+ **case.fields,
+ **extras,
+ }
+ for name in case.omit:
+ del body[name]
+ deltas = {"rollout-1/c1": kit.turn_1_tokens(case.turn_1_reply)}
+ # A disconnect cancels the native handler while the engine generates;
+ # Dynamo only marks the call's context stopped.
+ cancelled = ("running", asyncio.CancelledError()) if case.disconnect else None
+
+ async def both():
+ native = await _native(
+ kit,
+ *_host(kit, deltas, case.top_logprobs, cancelled or case.refuse),
+ body,
+ measure=False,
+ )
+ dynamo = await _dynamo(
+ kit,
+ *_host(kit, deltas, case.top_logprobs, case.refuse),
+ body,
+ extras,
+ stopped=case.disconnect,
+ )
+ return native, dynamo
+
+ native, dynamo = asyncio.run(both())
+
+ status = None if case.disconnect else case.status
+ assert (native.status, dynamo.status) == (status, status), (
+ native.reply,
+ dynamo.reply,
+ )
+ if status != 200:
+ # Only a call the client left reached the engine, with the same prompt.
+ assert [run[0] for run in native.runs] == [run[0] for run in dynamo.runs]
+ assert len(native.runs) == (1 if case.disconnect else 0)
+ # Capture is failed alike, and neither path staged the call.
+ assert native.failed == dynamo.failed == list(case.failed)
+ assert native.staged == dynamo.staged == []
+ # Gym is told alike that its capture call was refused before staging.
+ acknowledged = "c2" if case.acknowledged else None
+ assert native.acknowledged == dynamo.acknowledged == acknowledged
+ return
+ # The engine ran the same prompt, media and SamplingParams.
+ ((native_prompt, native_params, native_media),) = native.runs
+ ((dynamo_prompt, dynamo_params, dynamo_media),) = dynamo.runs
+ assert dynamo_prompt == native_prompt
+ _assert_same_media(dynamo_media, native_media)
+ assert (native_media is not None) == (case.kit == "vlm")
+ assert dynamo_params == native_params
+ # ...and was told the same of the call's reasoning, read from its template.
+ assert dynamo.reasoning_ended == native.reasoning_ended
+ if case.reasoning_ended is not None:
+ assert native.reasoning_ended == [case.reasoning_ended]
+ # Capture staged the same record and tensors, in the expected mode.
+ assert [record.mode for record, _ in native.staged] == (
+ [case.mode] if case.mode else []
+ )
+ assert len(dynamo.staged) == len(native.staged)
+ for (native_record, native_tensors), (dynamo_record, dynamo_tensors) in zip(
+ native.staged, dynamo.staged
+ ):
+ assert dynamo_record == native_record
+ assert (dynamo_tensors or {}).keys() == (native_tensors or {}).keys()
+ for name, tensor in (native_tensors or {}).items():
+ assert torch.equal(dynamo_tensors[name], tensor), name
+ assert ("routed_experts" in (native_tensors or {})) == (
+ case.kit == "router_replay"
+ )
+ assert dynamo.failed == native.failed == []
+ # Gym reads the same reply and token fields.
+ message = native.reply["choices"][0]["message"]
+ (choice,) = dynamo.reply["choices"]
+ assert choice["finish_reason"] == native.reply["choices"][0]["finish_reason"]
+ assert choice["delta"]["content"] == message["content"]
+ assert choice["delta"].get("reasoning_content") == message.get("reasoning")
+ assert [call["function"] for call in choice["delta"].get("tool_calls") or []] == [
+ call["function"] for call in message.get("tool_calls") or []
+ ]
+ result = dynamo.reply["nvext"]["nemo_rl"]
+ if native.staged:
+ assert result == {"ng_commit_coords": native.reply["ng_commit_coords"]}
+ else:
+ assert result == {
+ key: message[key]
+ for key in (
+ "prompt_token_ids",
+ "generation_token_ids",
+ "generation_log_probs",
+ )
+ }
+
+
+@pytest.mark.parametrize(
+ "observation",
+ [
+ OBSERVATION,
+ # Past the context window: Gym measures to decide whether to compact.
+ {"role": "user", "content": "And in Lyon?" + " Lyon" * MAX_MODEL_LEN},
+ ],
+ ids=["fits", "past_the_context"],
+)
+def test_a_measure_call_counts_the_prompt_the_call_would_run(kits, observation) -> None:
+ kit = kits("qwen3")
+ extras = {"required_prefix_token_ids": kit.turn_1_tokens()}
+ body = {
+ "model": kit.path,
+ "messages": [*HISTORY, observation],
+ "tools": TOOLS,
+ **extras,
+ }
+
+ async def both():
+ native = await _native(kit, *_host(kit, {}, 0), body, measure=True)
+ dynamo = await _dynamo(
+ kit,
+ *_host(kit, {}, 0),
+ {**body, "temperature": 0.7, "top_p": 0.9},
+ {**extras, "ng_measure": True},
+ )
+ return native, dynamo
+
+ native, dynamo = asyncio.run(both())
+
+ # Native Gym measures with /tokenize; through Dynamo, the chat call.
+ assert (native.status, dynamo.status) == (200, 200), dynamo.reply
+ assert (
+ dynamo.reply["nvext"]["nemo_rl"]["prompt_token_count"] == native.reply["count"]
+ )
+ assert dynamo.runs == []
+
+
+def _cut_call(
+ kit: _Kit, prompt_token_ids: list[int], saved: int, *, output_limit: int
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Cut a call that has generated ``saved`` of REPLY's tokens, as a checkpoint does.
+
+ Returns the cut's staged rows by key, and the continuation Gym puts in the
+ admission of the call it re-sends after a restore.
+ """
+ host, _, _ = _host(kit, {}, 0)
+ sink = _MemorySink()
+ host._capture_sink = sink
+ request = SimpleNamespace(ng_capture=_admission(mode="text"), stream=False)
+ VllmAsyncGenerationWorkerImpl._begin_request_capture(
+ host, request, prompt_token_ids
+ )
+ state = host._capture_registry.get(request)
+ state.set_effective_output_limit(output_limit)
+ reply_ids = kit.ids(REPLY)
+ state.record(
+ reply_ids[:saved],
+ [-0.5] * saved,
+ finish_reason="stop" if saved == len(reply_ids) else None,
+ )
+ (ack,) = (
+ _cuts(host, tokenizer=kit.tokenizer)
+ .checkpoint_cut(
+ GenerationCutInventory.build(
+ checkpoint_id="checkpoint-1",
+ server_name="policy_model",
+ active_prefixes=[
+ GenerationCutPrefix(
+ ticket_id="ticket-1",
+ rollout_id="rollout-1",
+ attempt=0,
+ model_call_id="c2",
+ admitted_at=1.0,
+ )
+ ],
+ )
+ )
+ .prefixes
+ )
+ assert ack.disposition == "durable_prefix"
+ (cut_key,) = ack.staging_keys
+ cut_record = sink.generation_prefix_records[-1][1]
+ return {cut_key: cut_record}, {
+ "source_capture_key": "rollout-1",
+ "source_model_call_id": "c2",
+ "staging_keys": [cut_key],
+ "prefix_token_count": ack.prefix_token_count,
+ "prefix_digest": cut_record.digest,
+ "effective_output_limit": ack.effective_output_limit,
+ "terminal_finish_reason": ack.terminal_finish_reason,
+ "terminal_stop_reason": ack.terminal_stop_reason,
+ }
+
+
+def _restored_host(kit: _Kit, records: dict[str, Any], resumed: int):
+ """A host after a restore: fresh, with the cut's rows in TQ."""
+ host, sink, failed = _host(kit, {}, 0)
+ host._capture_sink = _MemorySink()
+ _cuts(
+ host,
+ staging_source=_MemoryPrefixSource({}, records=records),
+ tokenizer=kit.tokenizer,
+ )
+ host.llm.resumed = resumed
+ return host, sink, failed
+
+
+def _reply_text(saved: int, kit: _Kit) -> str:
+ return kit.tokenizer.decode(kit.ids(REPLY)[:saved])
+
+
+@pytest.mark.parametrize(
+ ("cut_after", "rows_kept"),
+ [
+ # The cut lands inside the reasoning, ...
+ ("\nchecking", True),
+ # ... after it ended, so the reasoning parser must see the saved part, ...
+ ("\n\n", True),
+ # ... inside the tool call's JSON, ...
+ ('{"name": "get_weather"', True),
+ # ... or after the call had finished: nothing is left to decode.
+ (REPLY, True),
+ # A cut whose rows are gone is dropped, and the call regenerates.
+ ("\n\n", False),
+ ],
+ ids=["in_reasoning", "after_reasoning", "in_tool_call", "finished", "rows_gone"],
+)
+def test_a_call_cut_at_a_checkpoint_resumes_to_the_uninterrupted_reply(
+ kits, cut_after: str, rows_kept: bool
+) -> None:
+ kit = kits("qwen3")
+ reply_ids = kit.ids(REPLY)
+ # The cut saved the tokens up to the end of ``cut_after``.
+ saved = next(
+ count
+ for count in range(1, len(reply_ids) + 1)
+ if cut_after in _reply_text(count, kit)
+ )
+ body = {
+ "model": kit.path,
+ "messages": [*HISTORY, OBSERVATION],
+ "tools": TOOLS,
+ "temperature": 0.7,
+ "top_p": 0.9,
+ "max_tokens": 64,
+ "logprobs": True,
+ "top_logprobs": 0,
+ "ng_capture": _admission(mode="text"),
+ }
+ # Each path in a task of its own, as each request has on a server: a
+ # finished prefix is marked in a context variable.
+ whole = asyncio.run(_native(kit, *_host(kit, {}, 0), body, measure=False))
+ ((prompt, params, _),) = whole.runs
+ ((whole_record, _),) = whole.staged
+ records, continuation = _cut_call(
+ kit, prompt, saved, output_limit=params.max_tokens
+ )
+ resumed = saved if rows_kept else 0
+ extras = {
+ "ng_capture": CaptureAdmission(
+ capture_key="rollout-1-a1",
+ model_call_id="c3",
+ mode="text",
+ generation_cut=continuation,
+ ).model_dump(mode="json")
+ }
+ rows = records if rows_kept else {}
+ native = asyncio.run(
+ _native(
+ kit, *_restored_host(kit, rows, resumed), {**body, **extras}, measure=False
+ )
+ )
+ dynamo = asyncio.run(
+ _dynamo(kit, *_restored_host(kit, rows, resumed), {**body, **extras}, extras)
+ )
+
+ assert (native.status, dynamo.status) == (200, 200), (native.reply, dynamo.reply)
+ # The engine ran the saved rows as its prompt, with the budget that is left.
+ ((native_prompt, native_params, _),) = native.runs
+ ((dynamo_prompt, dynamo_params, _),) = dynamo.runs
+ assert native_prompt == dynamo_prompt == prompt + reply_ids[:resumed]
+ assert dynamo_params == native_params
+ if resumed < len(reply_ids):
+ assert native_params.max_tokens == params.max_tokens - resumed
+ # Capture staged one row for the call: the saved tokens, then the rest.
+ for outcome in (native, dynamo):
+ ((record, _),) = outcome.staged
+ assert record.token_ids_delta == whole_record.token_ids_delta
+ assert record.token_mask_delta == whole_record.token_mask_delta
+ assert (
+ record.generation_log_probs_delta == whole_record.generation_log_probs_delta
+ )
+ assert outcome.failed == []
+ # Gym reads the reply the uninterrupted call gave.
+ whole_choice = whole.reply["choices"][0]
+ native_choice = native.reply["choices"][0]
+ (dynamo_choice,) = dynamo.reply["choices"]
+ for choice, message in (
+ (native_choice, native_choice["message"]),
+ (dynamo_choice, dynamo_choice["delta"]),
+ ):
+ assert choice["finish_reason"] == whole_choice["finish_reason"]
+ assert message["content"] == whole_choice["message"]["content"]
+ assert (
+ message.get("reasoning", message.get("reasoning_content"))
+ == whole_choice["message"]["reasoning"]
+ )
+ assert [call["function"] for call in message["tool_calls"]] == [
+ call["function"] for call in whole_choice["message"]["tool_calls"]
+ ]
+ counts = ("prompt_tokens", "completion_tokens", "total_tokens")
+ for usage in (native.reply["usage"], dynamo.reply["usage"]):
+ assert [usage[count] for count in counts] == [
+ whole.reply["usage"][count] for count in counts
+ ]
diff --git a/tests/unit/models/generation/test_openai_server_utils.py b/tests/unit/models/generation/test_openai_server_utils.py
index 223be2dc883..36e7fcc910d 100644
--- a/tests/unit/models/generation/test_openai_server_utils.py
+++ b/tests/unit/models/generation/test_openai_server_utils.py
@@ -37,6 +37,7 @@ class _T:
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == template_token_ids
@@ -60,22 +61,7 @@ def decode(self, *args, **kwargs):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
- )
-
-
-def test_replace_prefix_tokens_tokenizer_without_eos_raises():
- """A tokenizer that has no EOS token cannot locate the splice boundary."""
-
- class _T:
- eos_token_id = None
-
- tokenizer = _T()
- with pytest.raises(AssertionError):
- replace_prefix_tokens(
- tokenizer=tokenizer,
- model_prefix_token_ids=[1],
- template_prefix_token_ids=[1, 2],
- template_token_ids=[1, 2],
+ eos_token_id=tokenizer.eos_token_id,
)
@@ -189,6 +175,7 @@ class _T:
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == [100, 2, 77, 88]
@@ -216,6 +203,7 @@ def decode(self, ids, **kwargs):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == [11, 12, 99, 99, 99, 55, 2, 70, 71, 2, 40, 41]
diff --git a/tests/unit/models/generation/test_vllm_captured_call_progress.py b/tests/unit/models/generation/test_vllm_captured_call_progress.py
index 675aef747c5..87e360ed415 100644
--- a/tests/unit/models/generation/test_vllm_captured_call_progress.py
+++ b/tests/unit/models/generation/test_vllm_captured_call_progress.py
@@ -65,7 +65,9 @@ async def exercise() -> None:
client = _AsyncLLMHTTPClient(
engine, asyncio.get_running_loop(), flat_logprobs=False
)
- sampling_params = SimpleNamespace(logprobs=None, prompt_logprobs=None)
+ sampling_params = SimpleNamespace(
+ logprobs=None, prompt_logprobs=None, extra_args=None
+ )
async def serve(model_call_id: Optional[str], request_id: str) -> None:
# What _begin_request_capture does in the request's task.
diff --git a/tests/unit/models/generation/test_vllm_chat_template_wiring.py b/tests/unit/models/generation/test_vllm_chat_template_wiring.py
index 5af1f45af01..3aaa08684df 100644
--- a/tests/unit/models/generation/test_vllm_chat_template_wiring.py
+++ b/tests/unit/models/generation/test_vllm_chat_template_wiring.py
@@ -37,8 +37,10 @@
import pytest
import torch
+from pydantic import BaseModel, ConfigDict
from nemo_rl.data.captured_media import capture_processed_media
+from nemo_rl.models.generation import vllm as vllm_generation_pkg
from nemo_rl.models.generation.prefix_recovery import NoPrefixRecovery
from nemo_rl.models.generation.vllm import utils as vllm_utils
from nemo_rl.models.generation.vllm.vllm_worker_async import (
@@ -51,6 +53,8 @@
# A class attribute would be shadowed by the subclass and the construction would
# be recorded somewhere the assertions never look.
_BUILT: dict[str, list] = {"renderer": [], "chat": [], "tokenize": []}
+_CHAT_SERVER = "nemo_rl.models.generation.vllm.chat_server"
+_PROMPT_AND_CAPTURE = "nemo_rl.models.generation.vllm.prompt_and_capture"
def _recorder(slot: str):
@@ -101,12 +105,25 @@ def decorator(fn):
def post(self, path, **_kwargs):
return self._register(path)
+ def exception_handler(self, _error_type):
+ return lambda fn: fn
+
def get(self, path, **_kwargs):
return self._register(path)
def _install_fake_vllm(monkeypatch):
"""Stub exactly the vLLM surface _setup_vllm_openai_api_server imports."""
+ # chat_server imports vLLM at module level: drop the cached copy (and the
+ # package attribute an earlier import set) so the server is built against
+ # these fakes; monkeypatch restores both afterwards.
+ for module_name, attribute in (
+ (_CHAT_SERVER, "chat_server"),
+ (_PROMPT_AND_CAPTURE, "prompt_and_capture"),
+ ):
+ monkeypatch.setitem(sys.modules, module_name, None)
+ del sys.modules[module_name]
+ monkeypatch.delattr(vllm_generation_pkg, attribute, raising=False)
for name in (
"vllm",
"vllm.entrypoints",
@@ -116,6 +133,7 @@ def _install_fake_vllm(monkeypatch):
"vllm.entrypoints.openai.models",
"vllm.entrypoints.serve",
"vllm.entrypoints.serve.engine",
+ "vllm.entrypoints.serve.exception_handling",
"vllm.entrypoints.serve.tokenize",
"vllm.entrypoints.serve.utils",
"vllm.reasoning",
@@ -141,7 +159,18 @@ def placeholder(name):
)
module(
"vllm.entrypoints.openai.chat_completion.protocol",
- ChatCompletionRequest=placeholder("ChatCompletionRequest"),
+ # A pydantic model that keeps unknown fields, as vLLM's request models are.
+ ChatCompletionRequest=type(
+ "ChatCompletionRequest",
+ (BaseModel,),
+ {
+ "model_config": ConfigDict(extra="allow"),
+ # What vLLM's rule for reasoning_ended reads off a request.
+ "__annotations__": {"include_reasoning": bool},
+ "include_reasoning": True,
+ "_grammar_from_parser": False,
+ },
+ ),
ChatCompletionResponse=placeholder("ChatCompletionResponse"),
)
module(
@@ -188,7 +217,20 @@ class ValidationError(ValueError):
def __init__(self, message, **kwargs):
super().__init__(message)
- module("vllm.exceptions", VLLMValidationError=ValidationError)
+ module(
+ "vllm.exceptions",
+ VLLMValidationError=ValidationError,
+ VLLMClientError=placeholder("VLLMClientError"),
+ GracefulHTTPError=placeholder("GracefulHTTPError"),
+ )
+ module(
+ "vllm.entrypoints.serve.exception_handling.error_response",
+ create_error_response=MagicMock(),
+ )
+ module(
+ "vllm.sampling_params",
+ StructuredOutputsParams=placeholder("StructuredOutputsParams"),
+ )
module(
"vllm.reasoning.abs_reasoning_parsers",
ReasoningParserManager=type(
@@ -311,6 +353,7 @@ async def test_captured_chat_logprobs_bypass(
top_logprobs=0,
# NeMoRLChatCompletionRequest's defaults for a call with no restored prefix.
_restored_generation_token_ids=(),
+ _restored_prompt_token_ids=(),
_restored_prefix_terminal=None,
return_tokens_as_token_ids=token_ids,
)
@@ -416,6 +459,7 @@ async def test_resumed_call_reports_usage_as_the_uninterrupted_call(monkeypatch)
top_logprobs=0,
# Prompt [1, 2, 3]; the call resumed from generated prefix [4, 5].
_restored_generation_token_ids=(4, 5),
+ _restored_prompt_token_ids=(),
_restored_prefix_terminal=None,
return_tokens_as_token_ids=False,
)
@@ -620,6 +664,7 @@ def install_with_real_response_type(monkeypatch):
for request in requests:
# NeMoRLChatCompletionRequest's defaults for a call with no restored prefix.
request._restored_generation_token_ids = ()
+ request._restored_prompt_token_ids = ()
request._restored_prefix_terminal = None
_, chats, _ = _build_server(monkeypatch, {}, capture_requests=[requests[0]])
chat = chats[0]
@@ -769,13 +814,6 @@ def capture(
"_resolve_admission_prefix",
lambda self, admission: [10, 77, 31, 2],
)
- monkeypatch.setattr(
- VllmAsyncGenerationWorkerImpl,
- "_enter_request_prefix",
- lambda self, request, prefix: setattr(
- request, "required_prefix_token_ids", prefix
- ),
- )
monkeypatch.setattr(
VllmAsyncGenerationWorkerImpl, "_capture_request_media", capture
)
@@ -785,6 +823,7 @@ def capture(
renderers, _, _ = _build_server(monkeypatch, {})
renderer = renderers[0]
renderer.renderer = SimpleNamespace(tokenizer=SimpleNamespace(eos_token_id=2))
+ renderer.model_config = SimpleNamespace(max_model_len=64)
request = SimpleNamespace(required_prefix_token_ids=[10, 77, 31, 2])
request.model_copy = lambda **kwargs: request
result = asyncio.run(
@@ -826,7 +865,14 @@ async def preprocess(self, *, messages, **kwargs):
VllmAsyncGenerationWorkerImpl, "_capture_media", False, raising=False
)
monkeypatch.setattr(
- VllmAsyncGenerationWorkerImpl, "_capture_admission", lambda self, request: None
+ VllmAsyncGenerationWorkerImpl,
+ "_capture_admission",
+ lambda self, request: SimpleNamespace(
+ mode="text", request_replay=None, generation_cut=None
+ ),
+ )
+ monkeypatch.setattr(
+ VllmAsyncGenerationWorkerImpl, "_begin_request_capture", MagicMock()
)
capture = MagicMock(
side_effect=ValueError("retiled media") if failure_stage == "capture" else None,
@@ -836,19 +882,21 @@ async def preprocess(self, *, messages, **kwargs):
monkeypatch.setattr(
VllmAsyncGenerationWorkerImpl, "_capture_request_media", capture
)
+ renderers, _, _ = _build_server(monkeypatch, {}, app=app)
+ # The server module was just imported against the fakes; patch that copy.
monkeypatch.setattr(
- "nemo_rl.models.generation.vllm.vllm_worker_async.remap_multimodal_placeholders",
- remap,
+ sys.modules[_PROMPT_AND_CAPTURE], "remap_multimodal_placeholders", remap
)
- renderers, _, _ = _build_server(monkeypatch, {}, app=app)
renderer = renderers[0]
renderer.renderer = SimpleNamespace(tokenizer=SimpleNamespace(eos_token_id=2))
renderer.model_config = SimpleNamespace(max_model_len=8 if overflows else 16)
endpoint = next(fn for path, fn in app.routes if path == "/v1/chat/completions")
- request = endpoint.__annotations__["request"]()
- request.required_prefix_token_ids = [10, 77, 31, 2]
- request.max_tokens, request.max_completion_tokens = 4, None
- request.model_copy = lambda **kwargs: request
+ request = endpoint.__annotations__["request"].model_construct(
+ messages=[],
+ required_prefix_token_ids=[10, 77, 31, 2],
+ max_tokens=4,
+ max_completion_tokens=None,
+ )
message = "maximum context length" if overflows else "retiled media"
with pytest.raises(ValueError, match=message):
@@ -922,13 +970,6 @@ async def preprocess(self, *, messages, **kwargs):
SimpleNamespace(token_ids_delta=delta, token_mask_delta=mask),
),
)
- monkeypatch.setattr(
- VllmAsyncGenerationWorkerImpl,
- "_enter_request_prefix",
- lambda self, request, prefix: setattr(
- request, "required_prefix_token_ids", prefix
- ),
- )
begin = MagicMock()
monkeypatch.setattr(VllmAsyncGenerationWorkerImpl, "_begin_request_capture", begin)
renderers, _, _ = _build_server(monkeypatch, {}, app=app)
@@ -936,9 +977,9 @@ async def preprocess(self, *, messages, **kwargs):
renderer.renderer = SimpleNamespace(tokenizer=SimpleNamespace(eos_token_id=2))
renderer.model_config = SimpleNamespace(max_model_len=64)
endpoint = next(fn for path, fn in app.routes if path == "/v1/chat/completions")
- request = endpoint.__annotations__["request"]()
- request.max_tokens, request.max_completion_tokens, request.min_tokens = 6, None, 4
- request.model_copy = lambda **kwargs: request
+ request = endpoint.__annotations__["request"].model_construct(
+ messages=[], max_tokens=6, max_completion_tokens=None, min_tokens=4
+ )
# vLLM may rebuild the prompt before it reaches the engine client.
built = []
diff --git a/tests/unit/models/generation/test_vllm_flat_logprobs.py b/tests/unit/models/generation/test_vllm_flat_logprobs.py
index f6f5db9003b..5388a375522 100644
--- a/tests/unit/models/generation/test_vllm_flat_logprobs.py
+++ b/tests/unit/models/generation/test_vllm_flat_logprobs.py
@@ -16,6 +16,7 @@
import pytest
+from nemo_rl.models.generation.vllm.utils import REASONING_ENDED_ARG
from nemo_rl.models.generation.vllm.vllm_worker_async import _AsyncLLMHTTPClient
pytestmark = pytest.mark.vllm
@@ -48,9 +49,26 @@ def test_generate_sets_flat_logprobs(
flat_logprobs, logprobs, prompt_logprobs, expected
):
sampling_params = SimpleNamespace(
- logprobs=logprobs, prompt_logprobs=prompt_logprobs, flat_logprobs=False
+ logprobs=logprobs,
+ prompt_logprobs=prompt_logprobs,
+ flat_logprobs=False,
+ extra_args=None,
)
_client(flat_logprobs=flat_logprobs).generate("prompt", sampling_params, "req-0")
assert sampling_params.flat_logprobs is expected
+
+
+def test_generate_hands_the_engine_the_template_reasoning_ended():
+ """vLLM's serving passes its own value, read from the spliced prompt."""
+ client = _client(flat_logprobs=False)
+ client._generate = lambda prompt, sampling_params, request_id, kwargs: kwargs
+ sampling_params = SimpleNamespace(
+ logprobs=None, extra_args={REASONING_ENDED_ARG: False, "other": 1}
+ )
+
+ kwargs = client.generate("prompt", sampling_params, "req-0", reasoning_ended=True)
+
+ assert kwargs == {"reasoning_ended": False}
+ assert sampling_params.extra_args == {"other": 1}
diff --git a/tests/unit/models/generation/test_vllm_generation.py b/tests/unit/models/generation/test_vllm_generation.py
index 4b37411baee..48c04fd2715 100644
--- a/tests/unit/models/generation/test_vllm_generation.py
+++ b/tests/unit/models/generation/test_vllm_generation.py
@@ -35,7 +35,8 @@
import requests
import torch
from fastapi import FastAPI, HTTPException, Request, Response
-from pydantic import ValidationError
+from fastapi.responses import StreamingResponse
+from pydantic import BaseModel, ConfigDict, ValidationError
from nemo_rl.algorithms.grpo import refit_policy_generation
from nemo_rl.algorithms.loss import NLLLossFn
@@ -710,7 +711,10 @@ def use_client_from_http_thread():
async def use_client():
tracing_enabled = await client.is_tracing_enabled()
outputs = [
- output async for output in client.generate(None, None, "request")
+ output
+ async for output in client.generate(
+ None, types.SimpleNamespace(extra_args=None), "request"
+ )
]
errored = client.errored
return (
@@ -783,7 +787,9 @@ async def abort(self, request_id):
def cancel_from_http_thread():
async def cancel():
async def consume():
- async for _ in client.generate(None, None, "cancelled-request"):
+ async for _ in client.generate(
+ None, types.SimpleNamespace(extra_args=None), "cancelled-request"
+ ):
pass
task = asyncio.create_task(consume())
@@ -1083,6 +1089,15 @@ def skip_fp8_known_failures() -> None:
def _install_fake_vllm_openai_modules(monkeypatch):
+ # chat_server imports vLLM at module level: drop the cached copy so the
+ # server is built against these fakes, and restore it afterwards.
+ for name in ("chat_server", "prompt_and_capture"):
+ module_name = f"nemo_rl.models.generation.vllm.{name}"
+ monkeypatch.setitem(sys.modules, module_name, None)
+ del sys.modules[module_name]
+ monkeypatch.delattr(
+ sys.modules["nemo_rl.models.generation.vllm"], name, raising=False
+ )
for module_name in (
"vllm",
"vllm.entrypoints",
@@ -1092,6 +1107,7 @@ def _install_fake_vllm_openai_modules(monkeypatch):
"vllm.entrypoints.openai.models",
"vllm.entrypoints.serve",
"vllm.entrypoints.serve.engine",
+ "vllm.entrypoints.serve.exception_handling",
"vllm.entrypoints.serve.tokenize",
"vllm.entrypoints.serve.utils",
"vllm.reasoning",
@@ -1210,7 +1226,18 @@ class ReasoningParserManager:
)
make_module(
"vllm.entrypoints.openai.chat_completion.protocol",
- ChatCompletionRequest=type("ChatCompletionRequest", (), {}),
+ # A pydantic model that keeps unknown fields, as vLLM's request models are.
+ ChatCompletionRequest=type(
+ "ChatCompletionRequest",
+ (BaseModel,),
+ {
+ "model_config": ConfigDict(extra="allow"),
+ # What vLLM's rule for reasoning_ended reads off a request.
+ "__annotations__": {"include_reasoning": bool},
+ "include_reasoning": True,
+ "_grammar_from_parser": False,
+ },
+ ),
ChatCompletionResponse=ChatCompletionResponse,
)
make_module(
@@ -1258,7 +1285,17 @@ def with_cancellation(handler):
"vllm.entrypoints.serve.tokenize.serving",
ServingTokenization=ServingTokenization,
)
- make_module("vllm.exceptions", VLLMValidationError=VLLMValidationError)
+ make_module(
+ "vllm.exceptions",
+ VLLMValidationError=VLLMValidationError,
+ VLLMClientError=type("VLLMClientError", (Exception,), {}),
+ GracefulHTTPError=type("GracefulHTTPError", (Exception,), {}),
+ )
+ make_module(
+ "vllm.entrypoints.serve.exception_handling.error_response",
+ create_error_response=MagicMock(),
+ )
+ make_module("vllm.sampling_params", StructuredOutputsParams=MagicMock())
make_module(
"vllm.reasoning.abs_reasoning_parsers",
ReasoningParserManager=ReasoningParserManager,
@@ -1282,6 +1319,9 @@ def decorator(func):
return decorator
+ def exception_handler(self, error_type):
+ return lambda func: func
+
def test_vllm_async_http_server_loads_reasoning_parser_plugin(monkeypatch):
(
@@ -1436,6 +1476,7 @@ def _fake_chat_request():
temperature=1.0,
max_tokens=1,
max_completion_tokens=None,
+ n=1,
# Read by NeMoRLOpenAIServingChatMixin.chat_completion_full_generator.
stream=False,
return_tokens_as_token_ids=None,
@@ -1444,6 +1485,7 @@ def _fake_chat_request():
# NeMoRLChatCompletionRequest's private defaults for a call that
# restores no generation prefix.
_restored_generation_token_ids=(),
+ _restored_prompt_token_ids=(),
_restored_prefix_terminal=None,
)
@@ -1499,9 +1541,12 @@ def test_chat_request_rejects_unreplayed_sampling_params(
request.top_k = None
setattr(request, field, value)
- # check_sampling_param raises ValueError for top_k; the rest are asserts.
- with pytest.raises((AssertionError, ValueError), match=field):
- asyncio.run(chat_handler(request, MagicMock()))
+ response = asyncio.run(chat_handler(request, MagicMock()))
+
+ # A client error Gym sees, not a server failure.
+ assert response.status_code == 400
+ assert field in json.loads(response.body)["error"]["message"]
+ serving_chat.create_chat_completion.assert_not_called()
def _track_request_capture(worker, request):
@@ -1514,6 +1559,29 @@ def _track_request_capture(worker, request):
return worker.token_capture, call
+def test_vllm_chat_endpoint_serves_a_streamed_call_and_fails_its_capture(
+ monkeypatch,
+) -> None:
+ """vLLM streams the reply, several choices included; capture cannot stage it."""
+ worker, serving_chat, chat_handler = _setup_fake_vllm_chat_handler(monkeypatch)
+
+ async def chunks():
+ yield "data: [DONE]\n\n" # pragma: no cover - never iterated
+
+ stream = chunks()
+ serving_chat.create_chat_completion = AsyncMock(return_value=stream)
+ request = _fake_chat_request()
+ request.stream = True
+ request.n = 2
+ token_capture, call = _track_request_capture(worker, request)
+
+ response = asyncio.run(chat_handler(request, MagicMock()))
+
+ assert isinstance(response, StreamingResponse)
+ assert response.body_iterator is stream
+ token_capture.fail_call.assert_called_once_with(call, reason="streaming_response")
+
+
def _stalled_engine_generator(started: asyncio.Event, cancelled: asyncio.Event):
"""Engine result generator for a generation that never finishes.
@@ -1558,16 +1626,18 @@ async def create_chat_completion(request, _raw_request):
@pytest.mark.asyncio
async def test_context_overflow_returns_http_400_for_nemo_gym(monkeypatch):
- """Overflow from _clamp_max_tokens must be HTTP 400 that NeMo Gym recognizes."""
+ """Overflow from clamp_max_tokens must be HTTP 400 that NeMo Gym recognizes."""
_, serving_chat, chat_handler = _setup_fake_vllm_chat_handler(monkeypatch)
+ prompt_and_capture = sys.modules[
+ "nemo_rl.models.generation.vllm.prompt_and_capture"
+ ]
max_model_len = 128
- renderer = serving_chat.kwargs["online_renderer"]
- renderer.model_config = types.SimpleNamespace(max_model_len=max_model_len)
- overflow_prompt = [0] * max_model_len
async def create_chat_completion(request, _raw_request):
- renderer._clamp_max_tokens(request, request.max_tokens, overflow_prompt)
+ prompt_and_capture.clamp_max_tokens(
+ request.max_tokens, max_model_len, max_model_len
+ )
serving_chat.create_chat_completion = create_chat_completion
response = await chat_handler(_fake_chat_request(), MagicMock())
@@ -1600,14 +1670,17 @@ async def create_chat_completion(request, _raw_request):
def test_clamp_max_tokens_sets_the_limit_the_request_uses(
monkeypatch, max_tokens, max_completion_tokens, expected
):
- _, serving_chat, _ = _setup_fake_vllm_chat_handler(monkeypatch)
- renderer = serving_chat.kwargs["online_renderer"]
- renderer.model_config = types.SimpleNamespace(max_model_len=2048)
+ _setup_fake_vllm_chat_handler(monkeypatch)
+ prompt_and_capture = sys.modules[
+ "nemo_rl.models.generation.vllm.prompt_and_capture"
+ ]
request = types.SimpleNamespace(
max_tokens=max_tokens, max_completion_tokens=max_completion_tokens
)
- renderer._clamp_max_tokens(request, 424, [0] * 1024)
+ prompt_and_capture.set_max_tokens(
+ request, prompt_and_capture.clamp_max_tokens(424, 1024, 2048)
+ )
assert vars(request) == expected
@@ -1615,10 +1688,7 @@ def test_clamp_max_tokens_sets_the_limit_the_request_uses(
@pytest.mark.asyncio
async def test_vllm_parser_does_not_see_parallel_tool_calls_false(monkeypatch):
"""vLLM 0.33 turns the flag into a grammar that masks sampling."""
- worker, serving_chat, _ = _setup_fake_vllm_chat_handler(monkeypatch)
- worker._capture_media = False
- worker._capture_request_media = MagicMock(return_value=None)
- worker._begin_request_capture = MagicMock()
+ _, serving_chat, _ = _setup_fake_vllm_chat_handler(monkeypatch)
renderer = serving_chat.kwargs["online_renderer"]
seen_by_vllm = []
@@ -1632,7 +1702,7 @@ async def preprocess_chat(self, request, **kwargs):
)
request = types.SimpleNamespace(parallel_tool_calls=False)
- await renderer.preprocess_chat(request, [], None, "auto", None)
+ await renderer._render_chat_call(request, [], {}, admission=None)
assert seen_by_vllm == [None]
# vLLM reads the flag again to trim the response to the first tool call.
@@ -1658,9 +1728,11 @@ def test_captured_requests_keep_vllm_output_kind(monkeypatch, stream, expected_k
raising=False,
)
request_cls = inspect.signature(chat_handler).parameters["request"].annotation
- request = request_cls.__new__(request_cls)
- request.stream = stream
- request.ng_capture = {"rollout_id": "r0", "model_call_id": "c1", "mode": "text"}
+ request = request_cls.model_construct(
+ messages=[],
+ stream=stream,
+ ng_capture={"rollout_id": "r0", "model_call_id": "c1", "mode": "text"},
+ )
assert request.to_sampling_params().output_kind == expected_kind
@@ -1938,12 +2010,22 @@ async def create_chat_completion(_request, _raw_request):
async def test_chat_handler_rejects_off_policy_sampling_and_pins_top_k(monkeypatch):
"""The route rejects off-policy sampling, then hands vLLM top_k=-1."""
_, serving_chat, chat_handler = _setup_fake_vllm_chat_handler(monkeypatch)
- serving_chat.create_chat_completion = AsyncMock(return_value=iter(()))
+ response_type = sys.modules[
+ "vllm.entrypoints.openai.chat_completion.protocol"
+ ].ChatCompletionResponse
+ serving_chat.create_chat_completion = AsyncMock(return_value=response_type())
+ # The server module was just imported against the fakes; patch that copy.
+ monkeypatch.setattr(
+ sys.modules["nemo_rl.models.generation.vllm.chat_server"],
+ "model_dump_chat_response_with_dynamic_message_fields",
+ lambda _response: {"choices": []},
+ )
off_policy = _fake_chat_request()
off_policy.temperature = 0.5
- with pytest.raises(ValueError, match="matches neither"):
- await chat_handler(off_policy, MagicMock())
+ response = await chat_handler(off_policy, MagicMock())
+ assert response.status_code == 400
+ assert "matches neither" in json.loads(response.body)["error"]["message"]
serving_chat.create_chat_completion.assert_not_awaited()
request = _fake_chat_request()
@@ -2004,8 +2086,9 @@ def test_chat_disconnect_releases_capture_queued_for_finalization(
"vllm.entrypoints.openai.chat_completion.protocol"
].ChatCompletionResponse
serving_chat.create_chat_completion = AsyncMock(return_value=response_type())
+ # The server module was just imported against the fakes; patch that copy.
monkeypatch.setattr(
- "nemo_rl.models.generation.vllm.vllm_worker_async."
+ sys.modules["nemo_rl.models.generation.vllm.chat_server"],
"model_dump_chat_response_with_dynamic_message_fields",
lambda _response: {"choices": []},
)
@@ -3207,14 +3290,19 @@ def test_vllm_validate_settings_accepts_missing_vllm_cfg_as_refit_disabled():
VllmGeneration.validate_settings(master_config)
-def _dynamo_master_config(*, fleet_health=False):
+def _dynamo_master_config(
+ *, router=False, fleet_health=False, gym=False, skip_tokenizer_init=False
+):
vllm_config = deepcopy(basic_vllm_test_config)
vllm_config["dynamo"] = True
vllm_config["vllm_cfg"]["async_engine"] = True
vllm_config["vllm_cfg"]["expose_http_server"] = True
+ vllm_config["vllm_cfg"]["skip_tokenizer_init"] = skip_tokenizer_init
return types.SimpleNamespace(
+ env={"should_use_nemo_gym": gym},
policy={"generation": vllm_config},
async_rl=types.SimpleNamespace(
+ generation_router=types.SimpleNamespace(enabled=router),
generation_fleet_health=types.SimpleNamespace(enabled=fleet_health),
),
)
@@ -3224,7 +3312,11 @@ def _dynamo_master_config(*, fleet_health=False):
("settings", "error"),
[
({}, None),
+ ({"router": True}, "routes the whole fleet behind one URL"),
({"fleet_health": True}, "Disable fleet health"),
+ # Gym's replies need the engine's tokenizer; NeMo RL's rollouts do not.
+ ({"gym": True, "skip_tokenizer_init": True}, "skip_tokenizer_init"),
+ ({"skip_tokenizer_init": True}, None),
],
)
def test_vllm_validate_settings_rejects_what_dynamo_cannot_serve(settings, error):
@@ -3236,6 +3328,69 @@ def test_vllm_validate_settings_rejects_what_dynamo_cannot_serve(settings, error
VllmGeneration.validate_settings(master_config)
+@pytest.mark.parametrize("announced", [True, False])
+def test_vllm_dynamo_chat_settings_reach_the_frontend_while_the_engines_load(
+ monkeypatch, announced: bool
+):
+ from ray.util.queue import Empty
+
+ from nemo_rl.models.generation.vllm import vllm_generation
+
+ events: list = []
+ settings = {"nemo_rl_chat": {"engine_args": {}}}
+
+ class _Queue:
+ def __init__(self, actor_options):
+ self.shut_down = False
+
+ def get(self, timeout):
+ if not announced:
+ raise Empty
+ return settings
+
+ def shutdown(self):
+ self.shut_down = True
+
+ monkeypatch.setattr(vllm_generation, "RayQueue", _Queue)
+ monkeypatch.setattr(vllm_generation.ray, "get", lambda refs: refs)
+ # A load that ends before any settings arrive: the driver stops waiting.
+ monkeypatch.setattr(
+ vllm_generation.ray, "wait", lambda refs, num_returns, timeout: (refs, [])
+ )
+ for step in (
+ "_post_init",
+ "_report_dp_openai_server_base_urls",
+ "_report_device_id",
+ "_register_dynamo_engines",
+ ):
+ monkeypatch.setattr(VllmGeneration, step, lambda self: None)
+ leader = types.SimpleNamespace(
+ announce_dynamo_chat_settings_to=types.SimpleNamespace(
+ remote=lambda queue: events.append(("announce to", queue))
+ )
+ )
+ generation = object.__new__(VllmGeneration)
+ generation.worker_group = types.SimpleNamespace(
+ workers=[leader],
+ get_dp_leader_worker_idx=lambda shard: 0,
+ run_all_workers_single_data=lambda method, **kwargs: (
+ events.append(method) or ["load"]
+ ),
+ )
+ generation._dynamo_runtime = types.SimpleNamespace(
+ serves_chat=True,
+ publish_chat_settings=lambda published: events.append(("publish", published)),
+ )
+
+ generation.load_and_start()
+
+ # The leader is asked before it loads, and the frontend gets what it sent.
+ (_, queue), load, *published = events
+ assert load == "load_model"
+ assert published == ([("publish", settings)] if announced else [])
+ assert queue.shut_down
+
+
def test_vllm_dynamo_rollouts_go_through_the_frontend(monkeypatch):
from nemo_rl.experience.failures import GenerationUnavailable
from nemo_rl.models.generation.dynamo import rollout_client
@@ -3277,6 +3432,8 @@ async def frontend(url, payload, timeout_s):
# The frontend request carries token IDs only, so media prompts stay native.
media = BatchedDataDict({**data, "vllm_content": [None]})
assert generation._dynamo_serves(data) and not generation._dynamo_serves(media)
+ # NeMo-Gym is pointed at the frontend too.
+ assert generation.client_base_urls == ["http://frontend/v1"]
_, output = asyncio.run(anext(generation.generate_async(data, greedy=True)))
@@ -4118,6 +4275,8 @@ def test_vllm_http_server(cluster, tokenizer):
top_p=generation_config["top_p"],
# We want to test the actual train flow and how this is used. So we need to get logprobs here.
logprobs=True,
+ # As NeMo-Gym sends it: the chosen token's logprob only.
+ top_logprobs=0,
max_tokens=1,
)
@@ -4332,6 +4491,7 @@ def test_vllm_deferred_model_load(cluster, tokenizer):
temperature=generation_config["temperature"],
top_p=generation_config["top_p"],
logprobs=True,
+ top_logprobs=0,
return_tokens_as_token_ids=True,
max_tokens=1,
)
@@ -4377,6 +4537,7 @@ def test_VllmAsyncGenerationWorker_replace_prefix_tokens(tokenizer):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == og_model_token_ids
@@ -4390,6 +4551,7 @@ def test_VllmAsyncGenerationWorker_replace_prefix_tokens(tokenizer):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == og_model_token_ids
@@ -4404,6 +4566,7 @@ def test_VllmAsyncGenerationWorker_replace_prefix_tokens(tokenizer):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == og_model_token_ids
@@ -4419,6 +4582,7 @@ def test_VllmAsyncGenerationWorker_replace_prefix_tokens(tokenizer):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == og_model_token_ids
@@ -4431,6 +4595,7 @@ def test_VllmAsyncGenerationWorker_replace_prefix_tokens(tokenizer):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == model_token_ids
@@ -4444,6 +4609,7 @@ def test_VllmAsyncGenerationWorker_replace_prefix_tokens(tokenizer):
model_prefix_token_ids=model_prefix_token_ids,
template_prefix_token_ids=template_prefix_token_ids,
template_token_ids=template_token_ids,
+ eos_token_id=tokenizer.eos_token_id,
)
assert result == model_token_ids
@@ -4486,6 +4652,7 @@ async def test_vllm_http_server_correct_merged_tokens_matches_baseline(
top_p=generation_config["top_p"],
# We want to test the actual train flow and how this is used. So we need to get logprobs here.
logprobs=True,
+ top_logprobs=0,
return_tokens_as_token_ids=True,
max_tokens=1,
)
diff --git a/tests/unit/models/generation/test_vllm_token_capture_hosting.py b/tests/unit/models/generation/test_vllm_token_capture_hosting.py
index e95974f1952..1a8d1d133c6 100644
--- a/tests/unit/models/generation/test_vllm_token_capture_hosting.py
+++ b/tests/unit/models/generation/test_vllm_token_capture_hosting.py
@@ -1542,6 +1542,22 @@ def test_generation_cut_past_its_deadline_fails_uncut_calls_without_staging(capl
assert _uncut_reasons(caplog) == {"control_deadline": 1}
+def test_a_cut_runs_to_the_deadline_its_caller_gives_it():
+ """The Dynamo frontend hands every engine the deadline it waits to itself."""
+ sink = _MemorySink()
+ worker = _worker_with_capture(sink)
+ _live_call(worker, [11, 12], [-0.1, -0.2], prompt_token_ids=(10,))
+ inventory = _single_call_inventory()
+
+ # This worker sets no control timeout of its own.
+ receipt = asyncio.run(
+ _cuts(worker).run_cut(inventory, deadline=time.monotonic() - 1.0)
+ )
+
+ assert [ack.disposition for ack in receipt.prefixes] == ["durable_failure"]
+ assert sink.generation_prefix_records == []
+
+
def _active_call_inventory(
worker, *calls: tuple[str, list[int]], output_limit: int | None = 16
) -> tuple[GenerationCutInventory, list]:
@@ -2461,9 +2477,12 @@ def replay(tokenizer, **kwargs):
)
# vLLM's processed tail: EOS and any matched stop string already stripped.
output = SimpleNamespace(token_ids=[14, 151645], text=" tail")
- resumed_worker._prefix_recovery.restore_reply_text(
+ resumed = resumed_worker._capture_registry.get(request)
+ restore_reply_text(
request,
SimpleNamespace(outputs=[output]),
+ prefix_token_ids=resumed.resumed_generation_token_ids,
+ engine_prompt_token_ids=resumed.prompt_token_ids,
tokenizer=_ByteTokenizer(),
)
(replay_kwargs,) = replays
@@ -2475,7 +2494,6 @@ def replay(tokenizer, **kwargs):
# A second checkpoint taken before the resumed call finishes must retain
# the original cut key and append the newly generated tail. Gym v2 stores
# this complete key list in the replacement attempt's model record.
- resumed = resumed_worker._capture_registry.get(request)
resumed.record([14], [-0.3])
second_receipt = _cuts(resumed_worker).checkpoint_cut(
_single_call_inventory("c2", checkpoint_id="checkpoint-2", attempt=1)
@@ -2594,7 +2612,7 @@ def _staging_chain_request(prev_len: int = 3) -> _FakeRequest:
def test_staging_chain_prefix_flows_through_adapter_and_begin_call():
- """The resolved prefix reaches both the engine request and capture admission."""
+ """The prefix resolved from the staging chain is the one begin_call checks."""
sink = _MemorySink()
worker = _worker_with_capture(sink)
source = _MemoryPrefixSource({"r0/c1": [10, 11], "r0/c2": [12]})
@@ -2604,7 +2622,6 @@ def test_staging_chain_prefix_flows_through_adapter_and_begin_call():
admission = worker._capture_admission(request)
prefix = worker._resolve_admission_prefix(admission)
- worker._enter_request_prefix(request, prefix)
VllmAsyncGenerationWorkerImpl._begin_request_capture(
worker,
request,
@@ -2618,8 +2635,6 @@ def test_staging_chain_prefix_flows_through_adapter_and_begin_call():
# Never patched back into the wire context.
assert request.ng_capture == context_before
assert admission.required_prefix_token_ids == []
- # enter_prefix is the production writer of the request field.
- assert request.required_prefix_token_ids == prefix
state = worker._capture_registry.get(request)
assert state.call.prefix_token_ids == prefix
assert state.prompt_token_ids == [10, 11, 12, 20]
diff --git a/tests/unit/single_controller/test_setup.py b/tests/unit/single_controller/test_setup.py
index 5c262a5f1b0..c8e11428314 100644
--- a/tests/unit/single_controller/test_setup.py
+++ b/tests/unit/single_controller/test_setup.py
@@ -1243,6 +1243,7 @@ def test_periodic_checkpointing_does_not_enable_gym_participants(self):
["http://generation"],
MagicMock(name="tokenizer"),
policy_api_key=None,
+ generation_cut_control_urls=None,
)
assert mock_spinup.call_args.kwargs["turn_checkpointing_enabled"] is False
@@ -1269,11 +1270,13 @@ def test_gym_spinup_passes_generation_prefix_cuts_explicitly(
["http://generation"],
MagicMock(name="tokenizer"),
policy_api_key=None,
+ generation_cut_control_urls=["http://generation-cuts"],
)
kwargs = mock_spinup.call_args.kwargs
assert kwargs["turn_checkpointing_enabled"] is True
assert kwargs["generation_prefix_cuts_enabled"] is generation_prefix_cuts
+ assert kwargs["generation_cut_control_urls"] == ["http://generation-cuts"]
# The flag is a typed keyword, not an undeclared key in the capture config.
assert "generation_prefix_cuts_enabled" not in kwargs["token_capture"]
@@ -2665,7 +2668,7 @@ def test_nemo_gym_wires_env_handle(self, patched_factories):
assert data_kwargs["is_vlm"] is True
mock_spinup.assert_called_once_with(
mc.env,
- base_urls=patched_factories["fake_gen"].dp_openai_server_base_urls,
+ base_urls=patched_factories["fake_gen"].client_base_urls,
model_name="test-model",
# Reaches the actor once, at spinup, rather than riding along with every
# run_rollouts call.
@@ -2674,6 +2677,10 @@ def test_nemo_gym_wires_env_handle(self, patched_factories):
use_fastokens=False,
turn_checkpointing_enabled=False,
generation_prefix_cuts_enabled=False,
+ # Where the generation backend takes Gym's checkpoint cuts.
+ generation_cut_control_urls=patched_factories[
+ "fake_gen"
+ ].generation_cut_control_urls,
# The vLLM server's bearer token, handed to the Gym model server.
policy_api_key=patched_factories["fake_gen"].http_server_api_key,
token_capture=None,
@@ -2907,6 +2914,39 @@ def test_token_capture_always_creates_finalizer_actor_pool(
mc.token_capture.control_timeout_s
)
+ def test_dynamo_token_capture_tells_gym_to_speak_dynamo(self, patched_factories):
+ """The same vLLM workers stage the calls; Gym reaches them through the frontend."""
+ mc = _make_master_config(backend="vllm")
+ mc.policy["generation"].update(
+ {
+ "model_name": "test-model",
+ "stop_strings": None,
+ "stop_token_ids": None,
+ "top_k": None,
+ "dynamo": True,
+ "vllm_cfg": {"async_engine": True, "expose_http_server": True},
+ }
+ )
+ mc.token_capture.enabled = True
+ mc.env = {"should_use_nemo_gym": True, "nemo_gym": {}}
+ patched_factories["setup_response_data"].return_value = (list(range(8)), None)
+
+ with (
+ patch.object(sc_setup_mod, "should_use_nemo_gym", return_value=True),
+ patch.object(
+ sc_setup_mod, "build_nemo_gym_actors", return_value=MagicMock()
+ ),
+ patch.object(sc_setup_mod, "validate_dataset_agent_coverage"),
+ patch.object(sc_setup_mod, "router_replay_enabled", return_value=False),
+ patch(
+ "nemo_rl.experience.rollout_reassembler_actor.create_rollout_reassembler_actors",
+ return_value=[MagicMock()],
+ ),
+ ):
+ setup_single_controller(mc, MagicMock(pad_token_id=9))
+
+ assert mc.token_capture.generation_backend == "dynamo"
+
def test_nemo_gym_coverage_failure_shuts_down_shards(self, patched_factories):
mc = _make_master_config(colocated=False, backend="vllm")
mc.policy["generation"]["model_name"] = "test-model"
diff --git a/tests/unit/test_recipes_and_test_suites.py b/tests/unit/test_recipes_and_test_suites.py
index 719eb493ce1..b9eaf38c19c 100644
--- a/tests/unit/test_recipes_and_test_suites.py
+++ b/tests/unit/test_recipes_and_test_suites.py
@@ -260,7 +260,7 @@ def test_all_recipe_yamls_accounted_for_in_test_suites(
)
-def test_nightly_compute_stays_below_5400_hours(nightly_test_suite, tracker):
+def test_nightly_compute_stays_below_5480_hours(nightly_test_suite, tracker):
command = f"DRYRUN=1 HF_HOME=... HF_DATASETS_CACHE=... CONTAINER= ACCOUNT= PARTITION= ./tools/launch {' '.join(nightly_test_suite)}"
print(f"Running command: {command}")
@@ -292,8 +292,8 @@ def test_nightly_compute_stays_below_5400_hours(nightly_test_suite, tracker):
f"Last line of output was not as expected: '{last_line}'"
)
total_gpu_hours = float(last_line.split(":")[-1].strip())
- assert total_gpu_hours <= 5400, (
- f"Total GPU hours exceeded 5400: {last_line}. We should revisit the test suites to reduce the total GPU hours."
+ assert total_gpu_hours <= 5480, (
+ f"Total GPU hours exceeded 5480: {last_line}. We should revisit the test suites to reduce the total GPU hours."
)
tracker.track("total_nightly_gpu_hours", total_gpu_hours)
diff --git a/tests/unit/utils/test_outdated_config_checks.py b/tests/unit/utils/test_outdated_config_checks.py
index 854dcc83758..67358a15467 100644
--- a/tests/unit/utils/test_outdated_config_checks.py
+++ b/tests/unit/utils/test_outdated_config_checks.py
@@ -431,8 +431,8 @@ def _generation_config(**generation) -> dict:
r"dynamo_cfg\.metrics_exclude_prefixes",
),
(
- {"dynamo_cfg": {"worker_args": {"extra_cli_args": []}}},
- r"dynamo_cfg\.worker_args was",
+ {"dynamo_cfg": {"worker_args": {"tool_call_parser": "hermes"}}},
+ r"dynamo_cfg\.worker_args was removed",
),
(
{"dynamo_cfg": {"frontend_args": {"tokenizer_cache": False}}},