Repository navigation
feat(dynamo)!: Dynamo 1.7 nightly, high session cap, frontends across nodes - #5115
Draft
jthomson04 wants to merge 30 commits into
Draft
jthomson04 wants to merge 30 commits into
jthomson04 wants to merge 30 commits into
Conversation
Dynamo 1.7 makes the KV router's session-affinity table size configurable (--router-session-affinity-max-entries); 1.5.1 caps it at 65,536 sessions. 1.7.0 is not on PyPI yet, so this pins a nightly until it is. The 1.7 API changes NeMo RL meets: - WorkerConfig replaces structural_tag_mode/scope/schema with one structural_tag dict; None leaves structural tags off. - /health lists discovery instances only with ?instances=true, so the engine wait asks for them. - DYN_ENABLE_EXPERIMENTAL_PARSERS_V2 is deprecated, and its 1.7 meaning is explicit v2 for every parser family, which refuses pairs with no v2 family (hermes, for one). DYN_PARSER_VERSION=auto keeps 1.5.1's behavior: the unified v2 parsers where the pair has a family, v1 otherwise. - The power-of-two router is gone. router_mode drops it, and a config that still names it fails at startup with the replacement. The lock also moves certifi (>= 2026.10.23) and adds redis, both ai-dynamo requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
dynamo_cfg.frontend_args.session_affinity_max_entries (default 16,777,216) sizes the KV router's session-affinity table through Dynamo 1.7's --router-session-affinity-max-entries. Dynamo's own default of 65,536 answers new sessions past it with HTTP 529, well below NT4-scale concurrency. The table allocates entries as sessions start and reaps them after the TTL, so a high cap costs memory only when sessions use it. Every frontend holds the whole fleet's sessions, so the cap covers all of them together. The flag rides with the TTL: affinity is off without one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
One frontend on the driver node runs out of CPU at NT4-scale concurrency. dynamo_cfg.frontend_args.num_frontends (default 1) starts that many dynamo.frontend processes, each in a small Ray actor pinned round-robin to the generation cluster's nodes (RayVirtualCluster.get_node_ids). etcd and NATS stay on the driver. More than one frontend adds --router-replica-sync, so each KV router sees the requests the others have in flight; every router already gets the engines' KV events, and session-affinity bindings sync whenever a TTL is set. The actor requests 0.001 CPU, so a hard node pin never waits on CPUs that placement-group bundles booked. Ray kills the frontend with the actor. NeMo-Gym gets every frontend URL and hashes each session to one of them; NeMo RL's own rollouts pick one at random. Engine registration waits until every frontend routes to every engine, a frontend that exits fails it at once, and shutdown reads each router's KV-event counts before stopping it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Some frontend settings exist only as environment variables. The tokenizer's prefix cache is one: DYN_TOKENIZER_CACHE_BYTES sizes it, and Dynamo defaults it to 1/16 of the cgroup memory limit (up to 8 GiB), or 64 MiB on a node with no limit. The runtime drops every inherited DYN_* variable, so nothing could set them. dynamo_cfg.frontend_args.extra_env reaches every frontend, as extra_cli_args does its argv, and may not override a variable the runtime manages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
ai-dynamo 1.7.0.dev20261106 is the first PyPI nightly with --router-session-affinity-max-entries (ai-dynamo/dynamo#18240), so a frontend started with a session TTL no longer exits on an unknown flag. Its requirements match the Nov 5 nightly's; the relock also settles every extra on one certifi. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…-1.7 Brings in NVIDIA-NeMo#4979 at 45add8b: the engine split into dynamo/engine_adapter.py and dynamo/vllm/*, NeMo-Gym chat served by NeMo RL's chat engine inside Dynamo's frontend, NVIDIA-NeMo#4905 with upstream/main, and Gym at 9aac609. Resolutions: - managed_runtime: each frontend still runs on a generation node in a FrontendHost actor, now launching nemo_rl.models.generation.dynamo.vllm.frontend. The token-capture config the chat engine reads is written by each host on its own node (publish_chat_capture fans out), since a file on the driver is not visible to frontends on other nodes. Render workers and the Ray address reach every frontend through its environment. - No Dynamo parser setting: chat is parsed by NeMo RL, so DYN_PARSER_VERSION goes with NVIDIA-NeMo#4979's PARSERS_V2 removal. - native_engine.py stays deleted; the new adapter passes no structural-tag fields, so this branch's WorkerConfig change for 1.7 is no longer needed. - config, exemplar and guide carry both sides' keys (session cap, num_frontends, extra_env; render_workers). extra_env's example is now DYN_TCP_LISTEN_BACKLOG: chat no longer goes through Dynamo's tokenizer, so its tokenizer cache does nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 9, 2026
…tor-concurrency Brings in NVIDIA-NeMo#5115 restacked on NVIDIA-NeMo#4979 at 45add8b (NeMo RL's chat engine in Dynamo's frontend, Gym at 9aac609). No conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Brings in c6a4c66 (the generation worker's logging stays under Dynamo's Worker) and aa4493e (Gym turn recovery runs behind Dynamo in the L1 overlay). The merge is clean; managed_runtime.py is untouched by NVIDIA-NeMo#4979. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
etcd, NATS and each frontend run in a session of their own, so a Ctrl-C at the driver's terminal reaches only the driver, and they were stopped only by shutdown() and an atexit fallback. A SIGKILL runs neither: killing the driver, or the raylet SIGKILLing the SingleControllerActor, left them running. Ray does not cover this. It kills a worker's child processes only when the worker exits through Ray (an exit or force-kill request, logged as "Try killing all child processes"); a worker SIGKILLed from outside, or by the raylet after a failed exit request, leaves them running. Each service now starts with a watchdog (service_watchdog.py): a stdlib-only process, in a session of its own, that holds the read end of a pipe whose only write end the starter holds. It reads EOF once the starter is gone, however it went, and then sends the service's process group SIGTERM, and SIGKILL after the same 15 s grace shutdown() gives. It checks the service's start time first, so it never signals a reused PID. On a clean shutdown the starter stops the service and then kills the watchdog. The watchdog is the service's sibling, not its parent. Ray's own reaping SIGKILLs a worker's direct children, so a wrapping parent would die there and orphan the service; as a sibling, Ray still kills the service itself. PR_SET_PDEATHSIG is not an option: it fires when the starting thread exits, and SingleController setup starts the services from a ThreadPoolExecutor. The frontends get a watchdog too: Ray kills a FrontendHost's frontend when it kills the actor (its owner died), but not when the actor itself is SIGKILLed. The turn-recovery test no longer kills phase one's services itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…-1.7 Brings in 59179cd: the render pool imports Gym's CaptureAdmission only for captured calls. The merge is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…fail with 500) into jthomson04/dynamo-1.7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…ith 500 on the Gym chat path) into jthomson04/dynamo-1.7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 10, 2026
…alidArgument -> 500) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…NVIDIA-NeMo#4905 + NVIDIA-NeMo#4506) into jthomson04/dynamo-1.7 Each frontend now takes NeMo-Gym's generation cuts on a port of its own. FrontendHost.start picks the cut port on its node beside the HTTP port and returns both URLs (FrontendUrls). ManagedDynamoRuntime.generation_cut_control_urls lists one per frontend_urls entry, in the same order, and VllmGeneration.generation_cut_control_urls passes them to Gym, which pairs each with its base_url. publish_chat_capture sends the cut's bearer token and timeout to every frontend's capture file with the rest of the capture config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 11, 2026
… cuts through every frontend) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…NVIDIA-NeMo#4905 squash) into jthomson04/dynamo-1.7 Two conflicts, both resolved by keeping both sides: - docs/guides/dynamo-generation.md: "Where the cut goes" keeps NVIDIA-NeMo#4979's deadline (each engine cuts up to that frontend's deadline, token_capture.control_timeout_s from the cut's arrival there) and this branch's one cut port per frontend, one per base_url. - tests/unit/models/generation/test_dynamo_generation_cut.py: imports only; NVIDIA-NeMo#4979's late-engine test runs through this branch's per-host capture publishing. With several frontends each frontend that takes a cut reads the timeout from its own node's capture pickle, which publish_chat_capture writes on every host, and so takes its own deadline. An engine runs the cuts from different frontends one after another, each against its own deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 12, 2026
…t-call steps and cut deadline) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… registration) into jthomson04/dynamo-1.7 One conflict, in the guide's startup list: NVIDIA-NeMo#5115's several-frontends wording is kept, and the new bullet says each frontend starts its render workers. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 12, 2026
…rs start at registration) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…ile the engines load) into jthomson04/dynamo-1.7 Conflicts, where this branch runs each frontend in a FrontendHost actor: - managed_runtime.py: FrontendHost.start takes serves_chat and names the chat-settings file in the frontend's environment; publish_chat_settings goes to every host, as publish_chat_capture does; both write through one helper. - test_dynamo_managed_runtime.py: the chat-settings checks move to the host's launch test, and the spread test checks that every host gets serves_chat and the settings. - dynamo-generation.md: the render-workers bullet, for every frontend. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 13, 2026
…rs built while the engines load) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…m pinned to the rebased dialect) into jthomson04/dynamo-1.7 No conflicts. uv.lock still matches pyproject.toml (uv 0.13.6, lock --check). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 13, 2026
… bump and the rebased Gym pin) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…s) into jthomson04/dynamo-1.7 Only the Gym submodule pin moves (234bb55da to 6dc68d2fa). No conflicts. uv.lock still matches pyproject.toml (uv 0.13.6, lock --check). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 13, 2026
… to Gym#4817's review fixes) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… sampling_overrides) into jthomson04/dynamo-1.7 No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 13, 2026
… startup check also covers sampling_overrides) into jthomson04/gym-actor-concurrency Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…ery in the Dynamo L1 list) into jthomson04/dynamo-1.7 No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
The managed services stop when their starter is SIGKILLed (d044de8), so phase one's kill leaves nothing to clean up, as in the turn test. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 13, 2026
…the prefix-recovery Dynamo run) into jthomson04/gym-actor-concurrency No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… prefix recovery) into jthomson04/dynamo-1.7 Three files conflicted: - uv.lock: re-resolved with uv 0.11.28, starting from NVIDIA-NeMo#4979's lock. Against that lock it moves ai-dynamo and ai-dynamo-runtime from 1.5.1 to 1.7.0.dev20261106, adds redis 8.1.0, and drops pydantic 2.13.0 and pydantic-core 2.46.0. Only ai-dynamo-runtime 1.5.1 pinned those two, so the lock no longer forks on them; most of the changed lines are that fork's markers going away. - docs/guides/dynamo-generation.md: this branch's wording for several frontends, with its sentences unwrapped as main's docs now are. - tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh: NVIDIA-NeMo#4979's side, which sources main's shared phase helper. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
NVIDIA-NeMo#4979 stops the Dynamo services that phase one's kill orphans from stop_phase1, which main has since moved into _gym_prefix_recovery_phases.sh. On this branch the services stop when their starter is killed (d044de8), so nothing is left to stop: the helper and the test are main's again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… engine and dynamic sampling under token capture) into jthomson04/dynamo-1.7 No conflicts. uv.lock is unchanged: the merged commits change no dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 15, 2026
…d multimodal prefix recovery) into jthomson04/gym-actor-concurrency No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 15, 2026
… engine and dynamic sampling under token capture) into jthomson04/gym-actor-concurrency No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…namic sampling for the streaming samplers and checkpointing configs as BaseModel) into jthomson04/dynamo-1.7 No conflicts. uv.lock is unchanged: the merged commits change no dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
added a commit
to jthomson04/RL
that referenced
this pull request
Nov 16, 2026
…namic sampling for the streaming samplers and checkpointing configs as BaseModel) into jthomson04/gym-actor-concurrency No conflicts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
… two-phase SimpleStorage checkpoint capture) into jthomson04/dynamo-1.7 No conflicts. uv.lock is unchanged: the merged commits change no dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
The last 1.7.0 nightly: Dynamo's main became 1.8.0 on Nov 16. Against dev20261106 it adds what NeMo RL has been working around: the Worker's logging opt-out (ai-dynamo/dynamo#18452), weight-free registration (#18295), named passthrough fields for Python chat processors (#18454) and custom chat processors (#18459). Its requirements equal dev20261106's, so the lock changes only the two packages. The 1.7.0 release branch has only the second of those, so the pin can no longer move to 1.7.0. It needs a release cut from main. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…ration Two workarounds go, now that the pinned Dynamo has the fixes: - The engine replaced `configure_dynamo_logging` with a no-op, because `Worker.run()` would take over the process's logging. It now passes `WorkerConfig(configure_logging=False)` (ai-dynamo/dynamo#18452). - The worker fetched a Hub ID's files without weights and registered that directory, because Dynamo fetched the whole checkpoint. Registration no longer fetches weights (ai-dynamo/dynamo#18295), so the engine registers its model name. The chat engine keeps `weight_free_model_path` for the render workers it builds before any engine registers. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Startup refused `stop_token_ids`, `allowed_token_ids`, `bad_words_token_ids`, `cache_salt`, `logprob_token_ids` and `detokenize` on the policy model with Dynamo, because the frontend accepted them and then dropped them before NeMo RL's chat engine. The pinned Dynamo hands them to a Python chat processor as top-level request fields (ai-dynamo/dynamo#18454), so they render as the native request fields they are and the refusal goes. `bad_words` is still refused: the frontend answers it with HTTP 400. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…processor `dynamo/vllm/frontend.py` replaced `dynamo.frontend.main.EntrypointArgs` to add NeMo RL's `chat_engine_factory`, and the chat engine read its settings from environment variables. Dynamo now loads a custom chat processor by name and hands it a JSON config (ai-dynamo/dynamo#18459), so: - every frontend starts with `--dyn-chat-processor` naming `chat_engine.provider`, and `--dyn-chat-processor-config` carrying the render workers, the Ray address and the capture and chat-settings files of its node; - the provider builds the factory from that config and, in a run that serves chat, starts the render workers at launch, as the wrapper did; - the `EntrypointArgs` wrap and the three environment variables go. `extra_cli_args` may not set the config flag either. `frontend.py` keeps only its `DistributedRuntime` wrap, which serves NeMo-Gym's generation cuts on a control port. That needs POST on Dynamo's frontend routes (ai-dynamo/dynamo#18492), which this nightly does not have yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…fusal header The chat engine raised `HttpError(status, message + suffix)` for a refused call, with Gym's capture acknowledgement appended to the message because a chat engine could not set a response header. On Dynamo 1.7 the frontend returns an `HttpError`'s status with the status's standard text, so Gym got a bare "Bad Request": it missed context-length refusals and their acknowledgements, and redispatched context-full rollouts. With the pinned Dynamo (ai-dynamo/dynamo#18459): - a 400 is raised as `InvalidArgument`, whose message the frontend returns; - an acknowledged capture refusal sets `x-ng-capture-refused` with `context.set_response_header`, the header the native route sends. Gym reads it as it does natively, and its message-suffix handling is no longer used; - an engine error Dynamo classified (`RoutedEngineError`: the engine is gone or overloaded) passes through with its own status. It was answered as 500. A refusal with another status still loses its message to the frontend's log. The engine keeps answering a refusal in its reply chunk: a raised error keeps its message only for a 400 and cannot carry the acknowledgement. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This draft fixes the two Dynamo limits we hit at NT4 scale (~0.5M rollouts in flight), and removes the workarounds in #4979 that Dynamo has since fixed. It is stacked on #4979.
--router-session-affinity-max-entries, feat(router): make the session-affinity entry limit configurable ai-dynamo/dynamo#18240). NeMo RL now sets it to 16,777,216 by default.num_frontendsstarts several frontends spread over the generation nodes, with--router-replica-syncwhen there is more than one.Commits
build(dynamo)!: move ai-dynamo to the 1.7.0 nightly. 1.7.0 isn't on PyPI yet; RC wheels exist only on internal Artifactory, so the pin is a PyPI nightly. Fixes for the 1.7 API:WorkerConfigtakes onestructural_tagdict instead ofstructural_tag_mode/scope/schema./healthlists instances only with?instances=true, so the engine wait asks for them.DYN_ENABLE_EXPERIMENTAL_PARSERS_V2=1now means explicit v2 for every parser family, which refuses pairs with no v2 family (hermes, for one).DYN_PARSER_VERSION=autokeeps 1.5.1's behavior: the unified v2 parsers where the pair has a family, v1 otherwise.power-of-tworouter.router_modedrops it, and a config that still names it fails at startup and names the replacement.certifi(>= 2026.10.23) and addsredis, both ai-dynamo requirements.feat(dynamo): raise the frontend's session-affinity cap. Addsfrontend_args.session_affinity_max_entries(default 16,777,216), passed with the TTL.feat(dynamo): run several frontends across the generation nodes. Addsfrontend_args.num_frontends(default 1).RayVirtualCluster.get_node_ids). etcd and NATS stay on the driver.feat(dynamo): pass extra environment to the frontends. Addsfrontend_args.extra_env, for settings Dynamo reads only from the environment.DYN_TOKENIZER_CACHE_BYTESsizes the tokenizer's prefix cache for NeMo-Gym chat. Dynamo's default is 1/16 of the cgroup memory limit, capped at 8 GiB, or 64 MiB with no limit.build(dynamo): pin the first nightly with the session-affinity cap. Moves the pin from1.7.0.dev20261105to1.7.0.dev20261106, the first PyPI nightly with feat(router): make the session-affinity entry limit configurable ai-dynamo/dynamo#18240. The earlier nightly rejected--router-session-affinity-max-entries, so a frontend started with a session TTL (the default) exited. The two nightlies have the same requirements; the relock settles every extra on onecertifi.build(dynamo): move ai-dynamo to the 1.7.0.dev20261116 nightly. The last 1.7 nightly: Dynamo'smainbecame 1.8.0 on Nov 16. Its requirements equaldev20261106's, so the lock changes only the two packages.refactor(dynamo): use Dynamo's logging opt-out and weight-free registration.configure_dynamo_loggingwith a no-op, becauseWorker.run()would take over the process's logging. It now passesWorkerConfig(configure_logging=False)(feat(backend): let embedded engines skip Dynamo's Python logging setup ai-dynamo/dynamo#18452).feat(dynamo): accept vLLM's token-level request fields from NeMo-Gym. Startup refusedstop_token_ids,allowed_token_ids,bad_words_token_ids,cache_salt,logprob_token_idsanddetokenizeon the policy model, because the frontend dropped them. It now hands them to a Python chat processor (fix(frontend): pass named passthrough fields to Python chat processors ai-dynamo/dynamo#18454).bad_wordsis still refused: the frontend answers it with HTTP 400.refactor(dynamo): load NeMo RL's chat engine as a Dynamo custom chat processor(feat(frontend): first-class custom chat processors ai-dynamo/dynamo#18459).--dyn-chat-processornaming NeMo RL's chat engine, and--dyn-chat-processor-configcarrying its settings.EntrypointArgsmonkeypatch and three environment variables go.dynamo/vllm/frontend.pykeeps only itsDistributedRuntimewrap, which serves generation cuts on a control port (see Before merge).fix(dynamo): answer refused Gym calls with their message and Gym's refusal header(feat(frontend): first-class custom chat processors ai-dynamo/dynamo#18459).HttpError's status with the status's standard text. Gym got a bare "Bad Request", missed context-length refusals and their acknowledgements, and redispatched context-full rollouts.InvalidArgument, whose message the frontend returns.x-ng-capture-refusedresponse header, as on the native route. Gym needs no change.Before merge
dynamo/vllm/frontend.py. That needs POST on Dynamo's frontend routes (feat(frontend): POST, request data and async handlers for Python route extensions ai-dynamo/dynamo#18492), which first ships in1.8.0.dev20261117.main.1.7.0cannot be the pin: its release branch has only #18240, #18295 and #18738 of the fixes this PR uses.vllm.general_pluginsentry point,dynamo_nixl_telemetry, so every vLLM process loads it. It imports onlyloggingandos, and it acts only whenNIXL_TELEMETRY_ENABLEis set. It is left as is.Testing
On
1.7.0.dev20261116(head50ffaab5b1, vLLM 0.33)Unit tests: vLLM lane 67 passed, vLLM + Gym lane 69, Gym lane 61, config tests 459, default lane 703 with the 10 local-only
nemo_automodelfailures intest_vllm_generation.py. ruff anduv lock --checkare clean; pyrefly shows only the 3 local errors atxferdtensor.py:196.Local 2-GPU functional runs. Each passes
check_dynamo_overlay.shand leaves no frontend, etcd or NATS process:grpo_dp_single_controller.shgrpo_async_gym_single_controller.shwith token capture,num_frontends=2grpo_async_gym_single_controller_prefix_recovery.shThe first prefix-recovery run passed in 1,085 s. The extra time was in the trainer (137 s of log-probability preparation, and about 200 s before each of two phases ended), and a rerun did not show it. The earlier head takes 565 s in the same setup.
Live probe of a running frontend, sent as NeMo-Gym's dialect sends them:
x-ng-capture-refused: <model_call_id>allowed_token_ids: [1000], 4 tokensEarlier, on
1.7.0.dev20261106Unit tests:
test_grpo.py,test_distillation.py,test_setup.pyand the virtual-cluster node-ID test;test_dynamo_*file;pyrefly, ruff and
uv lock --checkare clean.Local 2-GPU functional runs, with the default session TTL (300 s), so every frontend starts with
--router-session-affinity-max-entries 16777216. Each passescheck_dynamo_overlay.sh, and nodynamo.frontendprocess remains after shutdown:grpo_dp_single_controller.shgrpo_async_gym_single_controller.shwith token capture (hermes tool parser)num_frontends=2grpo_dp_single_controller_reload_refit.shWith two frontends, both start with
--router-replica-sync, both route to the engine, and both serve Gym calls. Ray folds the near-identical log lines of the two frontends, so the overlay check counts only 13 of the roughly 95 requests; the split is counted from the frontends' response lines, including Ray's folded repeats.Not covered here: frontends landing on different nodes, which needs a multi-node run.
🤖 Generated with Claude Code