Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
27791a4
feat(dynamo)!: serve any vLLM generation config through Dynamo's back…
jthomson04 Oct 24, 2026
08c50d2
fix(dynamo): address the #4905 review
jthomson04 Oct 25, 2026
2ba2cae
style(dynamo): sort the test's chat_routed_experts import
jthomson04 Oct 25, 2026
b065eb6
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 26, 2026
94c0e96
build(gym): pin Gym#4817 rebased onto Gym main
jthomson04 Oct 26, 2026
e182811
fix(dynamo): apply the Gym Dynamo dialect per actor, after shard comp…
jthomson04 Oct 26, 2026
b498510
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 27, 2026
db52655
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 27, 2026
9223ffb
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 27, 2026
b3cd664
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 28, 2026
e0d3789
refactor(vllm): share the per-sample rollout pipeline and chat token …
jthomson04 Oct 29, 2026
89469ca
Merge jthomson04/vllm-request-pipeline into jthomson04/dynamo-native-…
jthomson04 Oct 29, 2026
d217dd8
refactor(dynamo): split NeMo-Gym support out to its own PR
jthomson04 Oct 29, 2026
3f45420
feat(dynamo): serve NeMo-Gym through the Dynamo frontend
jthomson04 Oct 29, 2026
3928525
refactor(vllm): serve every rollout through one worker method
jthomson04 Oct 29, 2026
1a2a581
Merge jthomson04/vllm-request-pipeline into jthomson04/dynamo-native-…
jthomson04 Oct 29, 2026
2cfb157
Merge jthomson04/dynamo-native-engine into jthomson04/dynamo-gym
jthomson04 Oct 29, 2026
29d2004
fix: adopt Gym's capture_key rename and config paths for the Gym bump
jthomson04 Oct 29, 2026
46317dc
fix(dynamo): address the #4979 review and slim the NeMo-Gym path
jthomson04 Oct 29, 2026
a15ba5d
build: pin Gym to Gym#4817's head a7be5c5 (dynamo_worker pairing chec…
jthomson04 Oct 29, 2026
ed3ee35
refactor(dynamo): serve only NeMo RL rollouts; general OpenAI path mo…
jthomson04 Oct 29, 2026
83cab88
fix(dynamo): address round-2 review (C26, C28-C31, C33, C34, C36, C37…
jthomson04 Oct 29, 2026
1e73994
refactor(dynamo): debloat the Dynamo serving path
jthomson04 Oct 29, 2026
4aedf28
Revert "refactor(dynamo): serve only NeMo RL rollouts; general OpenAI…
jthomson04 Oct 29, 2026
3c8fd6b
Merge #4905's rollout-only split into jthomson04/dynamo-gym
jthomson04 Oct 29, 2026
b5e19b1
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 30, 2026
8a4ccd9
Merge jthomson04/dynamo-native-engine into jthomson04/dynamo-gym
jthomson04 Oct 30, 2026
23f13fa
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Oct 30, 2026
c60fc6d
Merge jthomson04/dynamo-native-engine into jthomson04/dynamo-gym
jthomson04 Oct 30, 2026
8ae17af
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Nov 1, 2026
c2e4ef9
Merge upstream/main into jthomson04/vllm-request-pipeline
jthomson04 Nov 2, 2026
6a02b99
Merge jthomson04/vllm-request-pipeline (#4978) into jthomson04/dynamo…
jthomson04 Nov 2, 2026
cf5b5bc
Merge jthomson04/dynamo-native-engine into jthomson04/dynamo-gym
jthomson04 Nov 2, 2026
580fe70
fix(dynamo): reject score centering at startup
jthomson04 Nov 2, 2026
0521cfc
Merge upstream/main into jthomson04/vllm-request-pipeline
jthomson04 Nov 3, 2026
e798c86
refactor(vllm): address review nits
jthomson04 Nov 3, 2026
7945239
Merge jthomson04/vllm-request-pipeline (#4978) into jthomson04/dynamo…
jthomson04 Nov 3, 2026
627f317
Merge jthomson04/dynamo-native-engine (#4905) into jthomson04/dynamo-gym
jthomson04 Nov 3, 2026
d67311f
style(vllm): sort check_sampling_param import
jthomson04 Nov 3, 2026
ea0ecb7
Merge jthomson04/vllm-request-pipeline (#4978) into jthomson04/dynamo…
jthomson04 Nov 3, 2026
f58d36f
Merge jthomson04/dynamo-native-engine (#4905) into jthomson04/dynamo-gym
jthomson04 Nov 3, 2026
2583db9
style(dynamo): sort capture-test imports
jthomson04 Nov 3, 2026
d15b6f4
test(vllm): expect ValueError from the chat handler's sampling check
jthomson04 Nov 4, 2026
b0b6b32
Merge jthomson04/vllm-request-pipeline (#4978) into jthomson04/dynamo…
jthomson04 Nov 4, 2026
9f8492e
Merge jthomson04/dynamo-native-engine (#4905) into jthomson04/dynamo-gym
jthomson04 Nov 4, 2026
1fe249f
Merge upstream/main into jthomson04/vllm-request-pipeline
jthomson04 Nov 4, 2026
44671d3
Merge jthomson04/vllm-request-pipeline (#4978) into jthomson04/dynamo…
jthomson04 Nov 4, 2026
9721dd4
Merge jthomson04/dynamo-native-engine (#4905) into jthomson04/dynamo-gym
jthomson04 Nov 4, 2026
f28b4d6
fix(dynamo): stage native route tensors from Gym capture
jthomson04 Nov 4, 2026
0b7d378
test(data-plane): bind _parse_capture_admission on the route-tensor f…
jthomson04 Nov 4, 2026
c4fe2bd
fix(dynamo): accept enable_auto_tools=false when no tool parser is set
jthomson04 Nov 5, 2026
7688abe
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Nov 5, 2026
4cfab93
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Nov 6, 2026
ee813c3
fix(dynamo): address round-3 review
jthomson04 Nov 6, 2026
614fb0a
test(dynamo): cover the round-3 gaps; trim stale docstrings
jthomson04 Nov 6, 2026
f75aa87
feat(dynamo): pin every NeMo-Gym session to one engine
jthomson04 Nov 6, 2026
776aaa3
feat(dynamo): tokenize Gym chat with fastokens in the frontend when v…
jthomson04 Nov 6, 2026
a1659e9
revert: drop the frontend fastokens flag and the every-call session pin
jthomson04 Nov 6, 2026
f774780
refactor(dynamo): split the engine adapter into a shared Dynamo side …
jthomson04 Nov 7, 2026
096e872
docs(dynamo): scope the Dynamo guide to the SingleController
jthomson04 Nov 7, 2026
6f6de14
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Nov 7, 2026
e506b68
Merge jthomson04/dynamo-native-engine (#4905) into jthomson04/dynamo-gym
jthomson04 Nov 7, 2026
3370812
refactor(dynamo): drop the endpoint_types setting; engines always ser…
jthomson04 Nov 7, 2026
790feb7
refactor(vllm): move the chat server out of the async worker
jthomson04 Nov 7, 2026
0b7894a
refactor(vllm): share the chat path's render, capture decision, turn …
jthomson04 Nov 7, 2026
43efaba
feat(dynamo)!: serve NeMo-Gym chat with NeMo RL's chat engine in Dyna…
jthomson04 Nov 7, 2026
817107f
build(gym): re-pin Gym to 4c84f173
jthomson04 Nov 8, 2026
a20b76d
test(dynamo): run every Gym overlay in fast CI; drop the swe1 recipe …
jthomson04 Nov 8, 2026
97e2a4b
docs(dynamo): review nits
jthomson04 Nov 8, 2026
9584986
fix(dynamo): acknowledge a refused capture call to Gym through the fr…
jthomson04 Nov 8, 2026
9331ee7
test(dynamo): add the step-order cases the review asked for
jthomson04 Nov 8, 2026
cca9af2
test(dynamo): drop the adapter test helper's unused prompt_logprobs
jthomson04 Nov 8, 2026
53a5298
fix(dynamo): answer a Gym call the engine refuses with the native status
jthomson04 Nov 8, 2026
6c56603
docs(dynamo): say what token_capture.generation_backend picks
jthomson04 Nov 8, 2026
62b375d
test(dynamo): fix the lint and Gym-dialect failures CI hit
jthomson04 Nov 9, 2026
fd91d76
fix(dynamo): fail engine errors and dead render workers as native does
jthomson04 Nov 9, 2026
3e81bca
fix(dynamo): render Gym calls as native does
jthomson04 Nov 9, 2026
f9172b4
fix(dynamo): reject at startup what the frontend would drop
jthomson04 Nov 9, 2026
53d7c9d
refactor(dynamo): one request contract, a typed plan, no dead code
jthomson04 Nov 9, 2026
28fd0e8
test(dynamo): tie the measure call to Gym; cover the budget fallback …
jthomson04 Nov 9, 2026
4fd57e8
docs(dynamo): correct the guide and comments
jthomson04 Nov 9, 2026
f7be6c1
test(dynamo): fake vLLM's chat request as a pydantic model in the wir…
jthomson04 Nov 9, 2026
45add8b
fix(dynamo): publish pydantic config blocks; keep the frontend host's…
jthomson04 Nov 9, 2026
c6a4c66
fix(dynamo): keep the generation worker's logging under Dynamo's Worker
jthomson04 Nov 9, 2026
5ff5507
Merge branch 'jthomson04/dynamo-native-engine' into jthomson04/dynamo…
jthomson04 Nov 9, 2026
aa4493e
test(dynamo): run Gym turn recovery behind Dynamo
jthomson04 Nov 9, 2026
59179cd
fix(dynamo): render calls without capture where NeMo-Gym is not insta…
jthomson04 Nov 9, 2026
4cbebb8
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Nov 10, 2026
a2c74e6
fix(dynamo): fail Gym calls with non-finite log-probs with 500, as na…
jthomson04 Nov 10, 2026
a9ff06b
fix(dynamo): answer an engine's InvalidArgument with 500 on the Gym c…
jthomson04 Nov 10, 2026
9919396
Merge upstream/main into jthomson04/dynamo-native-engine
jthomson04 Nov 11, 2026
726f7d2
Merge branch 'jthomson04/dynamo-native-engine' into jthomson04/dynamo…
jthomson04 Nov 11, 2026
e47ce35
feat(dynamo): cut running NeMo-Gym calls at a checkpoint
jthomson04 Nov 11, 2026
5c7fa69
feat(dynamo): resume a cut NeMo-Gym call from its saved tokens
jthomson04 Nov 11, 2026
b6ae1cf
Merge the #4905 squash (f488a0c6eb) into jthomson04/dynamo-gym
jthomson04 Nov 12, 2026
4f2ee77
Merge upstream/main into jthomson04/dynamo-gym
jthomson04 Nov 12, 2026
4002952
refactor(gym): share the prompt and capture steps of a Gym chat call …
terrykong Nov 11, 2026
7b536f5
fix(gym): run the shared chat-call steps under vLLM and Dynamo
jthomson04 Nov 12, 2026
f3d143b
refactor(gym): trim the shared chat-call steps
jthomson04 Nov 12, 2026
c810ee1
fix(vllm): answer vLLM errors raised while the request body is parsed
jthomson04 Nov 12, 2026
bbb35a2
test(gym): compare refused and failed calls on both paths
jthomson04 Nov 12, 2026
6877944
test(vllm): follow the shared chat-call checks in the native route tests
jthomson04 Nov 12, 2026
8415161
fix(gym): refuse stream: true on NeMo-Gym chat calls
jthomson04 Nov 12, 2026
7e29dcc
fix(gym): take reasoning_ended from the template render on both paths
jthomson04 Nov 12, 2026
4589ba2
fix(dynamo): give every engine the frontend's cut deadline
jthomson04 Nov 12, 2026
a6a13e7
docs(dynamo): the shared error map, refusals and the cut deadline
jthomson04 Nov 12, 2026
26d9d01
test: follow the shared chat-call steps outside the generation tests
jthomson04 Nov 12, 2026
56774f2
fix(dynamo): start the frontend's chat workers when the engines register
jthomson04 Nov 12, 2026
1cd7285
build(gym): bump the Gym submodule
zyzhou5 Nov 12, 2026
e1c510f
fix(dynamo): build the frontend's chat workers while the engines load
jthomson04 Nov 13, 2026
bb38b1d
Merge the Gym bump (#5195) and pin the rebased Dynamo dialect
jthomson04 Nov 13, 2026
34a41d8
build(gym): pin the Dynamo dialect with its review fixes
jthomson04 Nov 13, 2026
0d95075
fix(dynamo): reject request fields the frontend refuses or drops in s…
jthomson04 Nov 13, 2026
76111db
test(dynamo): run Gym prefix recovery behind Dynamo
jthomson04 Nov 13, 2026
766f81d
build(gym): pin Gym main, which has the Dynamo dialect
jthomson04 Nov 13, 2026
ffd880d
Merge main (e6d892c552: vLLM 0.33.0, docs reflow) into jthomson04/dyn…
jthomson04 Nov 13, 2026
011fe52
test(nightly): raise the GPU-hour cap to 5318 for the swe1 Dynamo nig…
jthomson04 Nov 13, 2026
4f97060
test(dynamo): give the render-pool test's parser the method vLLM 0.33…
jthomson04 Nov 14, 2026
e449a14
fix(gym): keep the native chat endpoint serving what it serves on main
jthomson04 Nov 14, 2026
7e4fbd5
Merge main (bb9ed784e4: multimodal prefix recovery) into jthomson04/d…
jthomson04 Nov 14, 2026
936f5d2
Merge main (705953be86: several HTTP frontends per engine, dynamic sa…
jthomson04 Nov 15, 2026
8904a7e
fix(dynamo): refuse several vLLM HTTP frontends behind Dynamo
jthomson04 Nov 15, 2026
027b75a
test(dynamo): run prefix recovery behind Dynamo in the fast set
jthomson04 Nov 15, 2026
4581367
Merge main (aa0f77af51: group length penalties, dynamic sampling for …
jthomson04 Nov 16, 2026
e3aec3d
Merge main (8f3595a112: Gym submodule bump with code_gen in OpenSandb…
jthomson04 Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion 3rdparty/Gym-workspace/Gym
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
Submodule Gym updated 542 files
61 changes: 52 additions & 9 deletions docs/design-docs/dynamo-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,33 @@
- A vLLM bump must never be blocked by Dynamo.
- Dynamo CI coverage must stay consistent with native vLLM's.

Both hold only if the engine process contains no Dynamo-owned vLLM code. A vLLM bump can then break Dynamo serving only in NeMo RL's own vLLM adapter (`vllm_adapter.py`), which calls a few vLLM APIs native NeMo RL does not: the `get_kv_cache_group_metadata` engine utility and `KVCacheSpecKind` at registration, and `StatLoggerBase` for the router's load signal. A renamed import fails the vLLM-lane unit tests at collection, and the Dynamo functional overlays run registration.
Both hold only if no process runs Dynamo-owned vLLM code. A vLLM bump can then break Dynamo serving only in NeMo RL's own `dynamo/vllm/` package:

- the engine adapter (`adapter.py`) calls a few vLLM APIs native NeMo RL does not: the `get_kv_cache_group_metadata` engine utility and `KVCacheSpecKind` at registration, and `StatLoggerBase` for the router's load signal;
- the frontend's chat engine (`chat_engine.py`, `render_pool.py`) builds the same vLLM renderer and chat serving the native server builds (`vllm/chat_server.py`), from the engine's own engine args.

A renamed import fails the vLLM-lane unit tests at collection, and the Dynamo functional overlays run registration and NeMo-Gym's chat calls.

## Engine hosting

`VllmGeneration` builds its workers exactly as without Dynamo. Each data-parallel leader (`VllmAsyncGenerationWorkerImpl`) owns an `AsyncLLM`. With Dynamo on, it also runs `dynamo.common.backend.Worker` as a task on its own actor event loop, which is the loop that owns the `AsyncLLM`.

`dynamo.common.backend` is Dynamo's engine-agnostic backend SDK and imports no vLLM. NeMo RL implements its `LLMEngine` in two parts:

- `dynamo/engine_adapter.py`: `DynamoEngineAdapter`, the Dynamo side shared by every generation backend. It serves request kinds as handlers keyed by their annotation, maps engine errors to a 400 that keeps the cause, and registers the endpoint for `/v1/completions` only. It imports no backend, and the engine only ever receives token IDs.
- `dynamo/vllm_adapter.py`: `DynamoVllmAdapter`, the vLLM backend over the worker's `AsyncLLM`. A backend implements `serve_rollout`, `start`, `abort` and `kv_event_sources`.
- `dynamo/engine_adapter.py`: `DynamoEngineAdapter`, the Dynamo side shared by every generation backend. It serves request kinds as handlers keyed by their annotation, maps engine errors to a 400 that keeps the cause, and registers the endpoint for `/v1/completions` (NeMo RL's rollouts) and chat (NeMo-Gym). It imports no backend, and the engine only ever receives token IDs.
- `dynamo/vllm/adapter.py`: `DynamoVllmAdapter`, the vLLM backend over the worker's `AsyncLLM`. A backend implements `serve_rollout`, `start`, `abort` and `kv_event_sources`, and may add request handlers.

The two together:

- `generate` serves only the rollouts NeMo RL itself sends, which carry an annotation, and refuses other requests. The worker's own `_serve_rollout` runs each rollout exactly as native `generate_async` does, and `request_pipeline.rollout_result` builds its result, routed experts included.
- `generate` maps Dynamo's token request onto vLLM `SamplingParams` the way vLLM's OpenAI server does, and streams token deltas back. Rollouts that NeMo RL itself sends carry an annotation. For those, the engine uses the same code as native `generate_async`:
- the worker's `_rollout_max_new_tokens` and `_rollout_sampling_params` build the request;
- `request_pipeline.rollout_result` builds the result, routed experts included.

The result travels in `engine_data`, and the driver assembles the same output with `request_pipeline.rollout_output`.

NeMo-Gym's calls arrive rendered and spliced by NeMo RL's chat engine (see [NeMo-Gym chat](#nemo-gym-chat)), annotated for the adapter's Gym handler, with the capture plan in `extra_args`. The handler runs the native server's steps after the render, with the worker's own helpers: the media snapshot (`_capture_request_media`), `begin_call` (which stamps the weight version), the generation with the `SamplingParams` vLLM's chat serving builds from the request, and the one TQ write. Its `engine_data` carries the output the frontend builds the reply from, and the call's `GymCallResult`. A call the engine refuses before it runs (a vLLM request error, a full queue, media capture refused) comes back in `engine_data` too, with the status the native server answers it with: Dynamo hands a raised error to the chat engine without its status.
- `abort` cancels the vLLM request on client disconnect.
- Registration advertises the context length, the KV block size (from vLLM's KV-cache group metadata), the block count and the scheduler limits. It publishes no token budget. The engine owns the context check and the output cap, as the native server does. With a budget, the frontend would refuse NeMo RL's own rollouts that fill the context.
- Registration advertises the context length, the KV block size (from vLLM's KV-cache group metadata), the block count and the scheduler limits. It publishes no token budget. The engine owns the context check and the output cap, as the native server does. With a budget, the frontend would fill an omitted `max_tokens` from the prompt it rendered, which a prefix splice then changes, and would refuse NeMo RL's own rollouts that fill the context.
- `kv_event_sources` points Dynamo at the engine's own vLLM KV-event ZMQ publisher. It is set only when the engine caches prefixes. Each engine binds its own Linux abstract-namespace socket (`ipc://@...`), so engines on one node need no port and cannot collide. The name is random rather than the pid, because containers that share the host network share these names. vLLM refuses `ipc://` only above data-parallel rank 0, and vLLM data parallelism is rejected at startup with Dynamo. A vLLM stat logger publishes the router's per-engine load signal.

Because the Worker runs on the actor's own loop, the worker's other methods interleave with Dynamo requests exactly as with native requests:
Expand All @@ -32,6 +42,38 @@ Colocated generation is rejected at startup: a sleeping engine stays registered,

No HTTP control routes are involved. The Rust Worker installs a SIGTERM handler that chains to Ray's, so actor shutdown is unchanged, and discovery drops an engine when its process exits.

## NeMo-Gym chat

Dynamo's frontend hands chat calls to a Python `chat_engine_factory` when one is given. `dynamo/vllm/frontend.py` starts `dynamo.frontend`'s own `main` with NeMo RL's factory added to the `EntrypointArgs` it builds, so the frontend keeps its HTTP server, router and `/v1/completions` path, and only chat goes to NeMo RL's chat engine (`dynamo/vllm/chat_engine.py`).

Each engine publishes, in its registration's `runtime_data`, its own engine args and the `policy.generation` chat settings. The chat engine builds a render pool (`dynamo/vllm/render_pool.py`) from them: `render_workers` spawned processes, each holding the vLLM renderer and chat serving the native server builds. Rendering is CPU-bound Python, and the frontend serves every call on one event loop, so it runs in the pool, first in first out.

When the engines expose chat (`vllm_cfg.expose_http_server`, which NeMo-Gym requires and the native worker starts its server on), the workers are built while the engines load, so NeMo-Gym's first calls do not wait for them:

- the frontend starts the workers at launch, and each imports vLLM;
- an engine sends the driver its settings as soon as it has its engine args, before it builds the engine. The driver hands them to the frontend in a file, as it does the capture config, and the workers build from them;
- at registration the chat engine keeps the workers if the engines register the same `policy.generation`, and replaces them otherwise. It does not compare the engine args, which vLLM adds to while it builds the engine (scheduler limits, resolved model paths; no render uses those). Workers that got no settings before registration build then.

A run without chat starts none. Per call:

1. the pool renders it with the native server's render step (`_render_chat_call`, shared with `vllm/chat_server.py`), including the token-capture decision for a Gym candidate, the history render the splice cuts against, and `reasoning_ended`, read from the template's render and never from a spliced prompt;
2. the chat engine checks on-policy sampling (`on_policy_sampling`, shared with the native route), reads a token-in prefix from TQ through a read-only client, splices it in with the model's turn-end tokens, and clamps the output budget (`prepare_prompt` in `vllm/prompt_and_capture.py`, shared with the native server);
3. the final token IDs go through the KV router to one engine, whose adapter runs the native server's `start_capture` and `save_capture`;
4. the pool builds the reply with vLLM's own response builder and parsers, through the same reply fixes for restored calls as the native server.

A call that is refused or fails is answered through one map on both paths (`error_reply` in `vllm/prompt_and_capture.py`): the status, the error body, the reason its capture is failed with, and whether Gym is told the capture call was refused before it was staged.

`GymCallExtras` and `GymCallResult` (`models/generation/gym_call.py`) define the fields both paths exchange with Gym. `tests/unit/models/generation/test_gym_chat_path.py` sends the same calls through the native routes and through this path and checks they match step for step.

### Prefix recovery

With `rollout_recovery.target_level: prefix`, a checkpoint cuts each running call (its decoded tokens are staged in TQ), and a restore resumes it from those tokens. The engines hold the calls and the worker-side state (`models/generation/prefix_recovery`), so both halves run there, with the native server's code:

- **The cut.** NeMo-Gym posts it beside the URL its calls go to, the frontend. A Dynamo frontend route can only be a synchronous GET, so `dynamo/vllm/frontend.py` also wraps `DistributedRuntime` and serves the native route (`vllm/generation_cut_route.py`) on a second port, in the frontend process (`dynamo/vllm/generation_cut.py`). It sends the cut to every engine with `Client.direct`; the adapter's `nemo_rl:generation_cut` handler runs the worker's own cut and answers with the calls it cut; and the answers are merged into the one receipt Gym expects. A call no engine cut gets Gym's failure ack and regenerates after a restore. The frontend takes one deadline when the cut arrives and sends it along, so an engine stops cutting in time for its answer to reach the frontend; an engine that timed the cut itself would start later and answer too late.
- **The resume.** The frontend renders and routes a re-sent call like any other. The adapter then runs the steps the native server runs (`start_capture` in `vllm/prompt_and_capture.py`): it reads and checks the saved rows, replaces the prompt with them, lowers the output budget, checks the call's reprocessed media against the digests the cut saved, and begins capture with the cut. A cut it cannot use (rows gone, or newer than its weights) is dropped, and the frontend's prompt runs. The engine's reply names what it resumed from, and the render pool puts the saved text back before vLLM's parsers run and counts the saved tokens as output (`prefix_recovery/restore.py`).

The resolve is in the engine, not the frontend, because the check needs the engine's weight version and its capture state, and a cut the engine drops then needs no second pass through the frontend. The cost is that the KV router scores a resumed call on the prompt without its saved tokens.

## Driver

`VllmGeneration` starts `ManagedDynamoRuntime`, which runs etcd, NATS and `dynamo.frontend`. After the engines load, it asks every DP leader to register and waits until the frontend routes to all of them. Each poll also asks the leaders whether their endpoint task stopped, so a failed registration fails with its cause instead of a timeout.
Expand All @@ -40,12 +82,13 @@ No HTTP control routes are involved. The Rust Worker installs a SIGTERM handler

## Dependency coupling

The vLLM workers' venv (the `vllm` extra) carries the released `ai-dynamo` and `ai-dynamo-runtime` wheels, never `ai-dynamo[vllm]`. The engine process only imports `dynamo.common.backend`, `dynamo.llm.exceptions` and `dynamo._core`.
The vLLM workers' venv (the `vllm` extra) carries the released `ai-dynamo` and `ai-dynamo-runtime` wheels, never `ai-dynamo[vllm]`. NeMo RL imports only the Dynamo modules `tests/unit/models/generation/test_dynamo_vllm_decoupling.py` lists: the backend SDK, `dynamo.llm` and the frontend's `main`, none of which import vLLM.

What a Dynamo upgrade has to re-check:
- the `dynamo.common.backend` contract (`LLMEngine`, `Worker`, `WorkerConfig`, publisher);
- the request fields of Dynamo's preprocessed request;
- the frontend's `nvext` extensions and annotation forwarding.
- the `dynamo.common.backend` contract (`LLMEngine`, `Worker`, `WorkerConfig`, publisher and logprob helpers);
- the request fields of Dynamo's preprocessed request, and `extra_args` reaching the engine;
- the frontend's `chat_engine_factory` hook and the `EntrypointArgs` name `dynamo/vllm/frontend.py` wraps, which are internal API;
- the Python chat request the hook receives, its `nvext` and annotation fields, and `EngineConfig.runtime_data` reaching the frontend.

The one vLLM↔Dynamo wire coupling is vLLM's KV-event msgpack, which Dynamo's router decodes. A change there degrades routing, not serving. The managed runtime logs the router's applied-event counts at shutdown, and the Dynamo functional overlays fail if the engines published events but none were applied (`tests/functional/check_dynamo_overlay.sh`).

Expand Down
Loading
Loading