Skip to content

Commit e182811

Browse files
jthomson04claude
andcommitted
fix(dynamo): apply the Gym Dynamo dialect per actor, after shard composition
From the #4905 review: - C23: setup_nemo_gym_config wrote the dialect as a top-level policy_model overlay, which a sharded Gym plan rejects as unowned. Setup now stashes it under a NeMo RL key (dynamo_dialect, alongside tokenizer_config), and _build_gym_actor_config applies it to each actor's composed config. A shard's own template kwargs still win over the native defaults. - C24: the local install instructions export NEMO_RL_DYNAMO_BIN_DIR, so the run that follows finds etcd and nats-server. - C25: the two in-function rationale blocks in native_engine.py are one line each; the design doc carries the full reasons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
1 parent 94c0e96 commit e182811

7 files changed

Lines changed: 158 additions & 45 deletions

File tree

‎docs/design-docs/dynamo-integration.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,12 +52,17 @@ implements its `LLMEngine` over the worker's `AsyncLLM`:
5252
- Registration advertises the context length, the KV block size (from vLLM's
5353
KV-cache group metadata), the block count and the scheduler limits. It
5454
publishes no token budget. The engine owns the context check and the output
55-
cap, because a prefix splice changes the prompt after the frontend has seen it.
55+
cap, as the native server does. With a budget, the frontend would fill an
56+
omitted `max_tokens` from the prompt it rendered, which a prefix splice then
57+
changes, and would refuse NeMo RL's own rollouts that fill the context.
5658
- `kv_event_sources` points Dynamo at the engine's own vLLM KV-event ZMQ
5759
publisher. It is set only when the engine caches prefixes. Each engine binds
5860
its own Linux abstract-namespace socket (`ipc://@...`), so engines on one node
59-
need no port and cannot collide. A vLLM stat logger publishes the router's
60-
per-engine load signal.
61+
need no port and cannot collide. The name is random rather than the pid,
62+
because containers that share the host network share these names. vLLM refuses
63+
`ipc://` only above data-parallel rank 0, and vLLM data parallelism is rejected
64+
at startup with Dynamo. A vLLM stat logger publishes the router's per-engine
65+
load signal.
6166

6267
Because the Worker runs on the actor's own loop, the worker's other methods
6368
interleave with Dynamo requests exactly as with native requests:

‎docs/guides/dynamo-generation.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,10 +173,12 @@ docker buildx build \
173173
.
174174
```
175175

176-
For a local checkout, install the binaries with:
176+
For a local checkout, install the binaries and keep the variable set for the
177+
runs that follow. The runtime looks for etcd and nats-server there, or else on `PATH`:
177178

178179
```bash
179-
NEMO_RL_DYNAMO_BIN_DIR=$PWD/venvs/dynamo-bin bash docker/dynamo/install.sh
180+
export NEMO_RL_DYNAMO_BIN_DIR="$PWD/venvs/dynamo-bin"
181+
bash docker/dynamo/install.sh
180182
```
181183

182184
## Run

‎nemo_rl/environments/nemo_gym.py‎

Lines changed: 45 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1897,27 +1897,14 @@ def setup_nemo_gym_config(config, tokenizer) -> None:
18971897
generation_config["stop_token_ids"] = None
18981898

18991899
if dynamo_enabled(generation_config):
1900-
# Gym's policy model calls the Dynamo frontend, which takes token data in
1901-
# nvext (Gym's Dynamo dialect) instead of vLLM's request extensions.
1902-
vllm_model = (
1903-
config.env.setdefault("nemo_gym", {})
1904-
.setdefault("policy_model", {})
1905-
.setdefault("responses_api_models", {})
1906-
.setdefault("vllm_model", {})
1907-
)
1908-
vllm_model["dynamo_frontend"] = True
1909-
# The frontend reads `reasoning` as another name for `reasoning_content` and
1910-
# refuses an echoed assistant turn that carries both, Gym's default.
1911-
vllm_model.setdefault("reasoning_field", "reasoning_content")
1912-
# The native server merges these under every request's own template
1913-
# kwargs; the frontend has no such default, so Gym sends them. They sit
1914-
# under the kwargs Gym sends: vllm_model's own, else extra_body's.
1915-
default_kwargs = native_default_chat_template_kwargs(generation_config)
1916-
if default_kwargs:
1917-
gym_kwargs = vllm_model.get("chat_template_kwargs") or (
1918-
(vllm_model.get("extra_body") or {}).get("chat_template_kwargs") or {}
1900+
# Gym's policy model calls the Dynamo frontend in Gym's Dynamo dialect.
1901+
# Each actor applies it to its own composed config (_apply_dynamo_dialect),
1902+
# so it never becomes a policy_model overlay that a shard must claim.
1903+
config.env.setdefault("nemo_gym", {})["dynamo_dialect"] = {
1904+
"default_chat_template_kwargs": native_default_chat_template_kwargs(
1905+
generation_config
19191906
)
1920-
vllm_model["chat_template_kwargs"] = {**default_kwargs, **gym_kwargs}
1907+
}
19211908

19221909
# For VLM runs, plumb the tokenizer config into the gym env config so the
19231910
# NemoGym actor can reconstruct the processor inside itself (needed for
@@ -1987,6 +1974,39 @@ def build_nemo_gym_config(
19871974
)
19881975

19891976

1977+
def _apply_dynamo_dialect(
1978+
nemo_gym_dict: dict[str, Any], default_chat_template_kwargs: Mapping[str, Any]
1979+
) -> None:
1980+
"""Point one actor's policy model at the Dynamo frontend, in Gym's Dynamo dialect.
1981+
1982+
The frontend takes token data in nvext instead of vLLM's request extensions.
1983+
Applied to the actor's composed config (a shard's overrides included), so
1984+
settings the user wrote there win. Copies the policy_model levels it
1985+
changes, so the caller's config is not mutated.
1986+
"""
1987+
policy_model = dict(nemo_gym_dict.get("policy_model") or {})
1988+
models = dict(policy_model.get("responses_api_models") or {})
1989+
vllm_model = dict(models.get("vllm_model") or {})
1990+
vllm_model["dynamo_frontend"] = True
1991+
# The frontend reads `reasoning` as another name for `reasoning_content` and
1992+
# refuses an echoed assistant turn that carries both, Gym's default.
1993+
vllm_model.setdefault("reasoning_field", "reasoning_content")
1994+
# The native server merges these under every request's own template
1995+
# kwargs; the frontend has no such default, so Gym sends them. They sit
1996+
# under the kwargs Gym sends: vllm_model's own, else extra_body's.
1997+
if default_chat_template_kwargs:
1998+
gym_kwargs = vllm_model.get("chat_template_kwargs") or (
1999+
(vllm_model.get("extra_body") or {}).get("chat_template_kwargs") or {}
2000+
)
2001+
vllm_model["chat_template_kwargs"] = {
2002+
**default_chat_template_kwargs,
2003+
**gym_kwargs,
2004+
}
2005+
models["vllm_model"] = vllm_model
2006+
policy_model["responses_api_models"] = models
2007+
nemo_gym_dict["policy_model"] = policy_model
2008+
2009+
19902010
def _build_gym_actor_config(
19912011
nemo_gym_dict: dict[str, Any],
19922012
*,
@@ -2015,6 +2035,11 @@ def _build_gym_actor_config(
20152035
invalid_tool_call_patterns = nemo_gym_dict.pop("invalid_tool_call_patterns", None)
20162036
thinking_tags = nemo_gym_dict.pop("thinking_tags", None)
20172037
tokenizer_config = nemo_gym_dict.pop("tokenizer_config", None)
2038+
dynamo_dialect = nemo_gym_dict.pop("dynamo_dialect", None)
2039+
if dynamo_dialect is not None:
2040+
_apply_dynamo_dialect(
2041+
nemo_gym_dict, dynamo_dialect["default_chat_template_kwargs"]
2042+
)
20182043
external_service_readiness = extract_external_service_readiness(nemo_gym_dict)
20192044
port_range = {
20202045
key: value

‎nemo_rl/environments/nemo_gym_shards.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,12 @@
3838
# Dict-shaped NeMo RL integration settings that are not Gym server entries.
3939
# The actor factory or rollout code consumes these separately.
4040
NEMO_RL_DICT_CONFIG_KEYS = frozenset(
41-
{"effort_levels", "external_service_readiness", "tokenizer_config"}
41+
{
42+
"dynamo_dialect",
43+
"effort_levels",
44+
"external_service_readiness",
45+
"tokenizer_config",
46+
}
4247
)
4348

4449
# Ray placement-group strategies a shard plan may ask for. STRICT_SPREAD is the

‎nemo_rl/models/generation/dynamo/native_engine.py‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -149,11 +149,7 @@ def dynamo_kv_events_config(
149149
return {
150150
"enable_kv_cache_events": True,
151151
"publisher": "zmq",
152-
# A Linux abstract-namespace socket (no port, no file): vLLM binds any
153-
# ipc:// endpoint and this process's Dynamo worker connects to it, so
154-
# engines on one node cannot collide. Random rather than the pid, since
155-
# containers sharing the host network share these names. vLLM rejects
156-
# ipc:// only at data_parallel_rank != 0, which Dynamo rejects at startup.
152+
# A random abstract-namespace socket per engine (dynamo-integration.md).
157153
"endpoint": f"ipc://@nemo-rl-kv-events-{os.urandom(8).hex()}",
158154
}
159155

@@ -363,11 +359,7 @@ async def start(self, worker_id: int) -> EngineConfig:
363359
return EngineConfig(
364360
model=model,
365361
served_model_name=model,
366-
# No token budget: the engine owns prompt overflow and the output
367-
# cap, as the native server does. A budget would make the frontend
368-
# fill an omitted max_tokens from the prompt it rendered, which a
369-
# NeMo-Gym prefix splice then changes, and refuse NeMo RL's own
370-
# rollouts that fill the context.
362+
# No token budget: the engine owns overflow (dynamo-integration.md).
371363
llm=LlmRegistration(
372364
context_length=self._max_model_len,
373365
kv_cache_block_size=block_size,

‎tests/unit/environments/test_nemo_gym_token_capture.py‎

Lines changed: 27 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -604,12 +604,11 @@ def test_setup_nemo_gym_config_vllm_speaks_dynamo_dialect_only_with_dynamo(
604604
assert vllm_cfg["async_engine"] is True
605605
assert vllm_cfg["expose_http_server"] is True
606606
assert should_use_nemo_gym(config) is True
607-
policy_model = config.env["nemo_gym"].get("policy_model") if dynamo else None
608607
if dynamo:
609-
assert policy_model["responses_api_models"]["vllm_model"] == {
610-
"dynamo_frontend": True,
611-
# The frontend refuses reasoning under both of Gym's keys.
612-
"reasoning_field": "reasoning_content",
608+
# A NeMo RL key each actor applies to its composed config, not a
609+
# policy_model overlay a sharded plan would need an owner for.
610+
assert config.env["nemo_gym"] == {
611+
"dynamo_dialect": {"default_chat_template_kwargs": {}}
613612
}
614613
else:
615614
assert "nemo_gym" not in config.env
@@ -633,12 +632,17 @@ def test_setup_nemo_gym_config_vllm_speaks_dynamo_dialect_only_with_dynamo(
633632
),
634633
],
635634
)
636-
def test_setup_nemo_gym_config_dynamo_sends_the_native_default_template_kwargs(
635+
def test_gym_actor_config_dynamo_sends_the_native_default_template_kwargs(
637636
vllm_model: dict, expected: dict
638637
) -> None:
639638
"""The native server's default template kwargs sit under the ones Gym sends."""
640639
from nemo_rl.algorithms.grpo import MasterConfig
641-
from nemo_rl.environments.nemo_gym import setup_nemo_gym_config
640+
from nemo_rl.environments.nemo_gym import (
641+
build_nemo_gym_config,
642+
setup_nemo_gym_config,
643+
)
644+
645+
user_vllm_model = copy.deepcopy(vllm_model)
642646

643647
config = MasterConfig.model_construct(
644648
env={
@@ -665,5 +669,19 @@ def test_setup_nemo_gym_config_dynamo_sends_the_native_default_template_kwargs(
665669
)
666670

667671
setup_nemo_gym_config(config, tokenizer=None)
668-
669-
assert vllm_model["chat_template_kwargs"] == expected
672+
gym_config = build_nemo_gym_config(
673+
config.env,
674+
base_urls=["http://frontend"],
675+
model_name="test-model",
676+
enable_router_replay=False,
677+
use_fastokens=False,
678+
)["initial_global_config_dict"]
679+
680+
composed = gym_config["policy_model"]["responses_api_models"]["vllm_model"]
681+
assert composed["dynamo_frontend"] is True
682+
# The frontend refuses reasoning under both of Gym's keys.
683+
assert composed["reasoning_field"] == "reasoning_content"
684+
assert composed["chat_template_kwargs"] == expected
685+
assert "dynamo_dialect" not in gym_config
686+
# The user's overlay is left as written.
687+
assert vllm_model == user_vllm_model

‎tests/unit/environments/test_nemo_gym_utils.py‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -932,6 +932,72 @@ def test_build_nemo_gym_actors_hands_every_actor_the_policy_api_key(
932932
assert all(cfg["policy_api_key"] == "per-job-key" for cfg in cluster.actor_configs)
933933

934934

935+
@pytest.mark.parametrize(
936+
"env_configs", [_env_configs, _shard_env_configs], ids=["unsharded", "sharded"]
937+
)
938+
def test_build_nemo_gym_actors_gives_every_actor_the_dynamo_dialect(
939+
detected_uv_dirs, env_configs
940+
):
941+
"""With Dynamo, each actor's composed config speaks Gym's Dynamo dialect.
942+
943+
The dialect is not a policy_model overlay, which a sharded plan would reject
944+
as unowned. A shard's own template kwargs still win over the native defaults.
945+
"""
946+
from nemo_rl.algorithms.grpo import MasterConfig
947+
948+
env = env_configs()
949+
sharded = "shards" in env["nemo_gym"]
950+
if sharded:
951+
env["nemo_gym"]["shards"][1]["overrides"] = {
952+
"policy_model": {
953+
"responses_api_models": {
954+
"vllm_model": {"chat_template_kwargs": {"enable_thinking": False}}
955+
}
956+
}
957+
}
958+
config = MasterConfig.model_construct(
959+
env=env,
960+
policy={
961+
"generation": {
962+
"backend": "vllm",
963+
"dynamo": True,
964+
"vllm_cfg": {
965+
"http_server_serving_chat_kwargs": {
966+
"default_chat_template_kwargs": {"enable_thinking": True}
967+
}
968+
},
969+
}
970+
},
971+
)
972+
setup_nemo_gym_config(config, tokenizer=None)
973+
cluster = _FakeGymCluster()
974+
975+
with _patched_cluster(cluster):
976+
nemo_gym_mod.build_nemo_gym_actors(
977+
config.env,
978+
base_urls=["http://frontend"],
979+
model_name="test-model",
980+
tokenizer=_TOKENIZER,
981+
enable_router_replay=False,
982+
use_fastokens=False,
983+
)
984+
985+
thinking = [
986+
cfg["initial_global_config_dict"]["policy_model"]["responses_api_models"][
987+
"vllm_model"
988+
]["chat_template_kwargs"]["enable_thinking"]
989+
for cfg in cluster.actor_configs
990+
]
991+
# judged, then the tools shard's two replicas with their own override.
992+
assert thinking == ([True, False, False] if sharded else [True])
993+
for cfg in cluster.actor_configs:
994+
gym_config = cfg["initial_global_config_dict"]
995+
vllm_model = gym_config["policy_model"]["responses_api_models"]["vllm_model"]
996+
assert vllm_model["dynamo_frontend"] is True
997+
assert vllm_model["reasoning_field"] == "reasoning_content"
998+
assert "dynamo_dialect" not in gym_config
999+
1000+
9351001
def test_build_nemo_gym_actors_spreads_every_replica_onto_its_own_node(
9361002
detected_uv_dirs,
9371003
):

0 commit comments

Comments
 (0)