Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
305608c to
f3fd7e2
Compare
a265e75 to
eb743cb
Compare
f3fd7e2 to
37213bc
Compare
37213bc to
74c9462
Compare
19bd47b to
5dae5e8
Compare
d4290d4 to
fdf1938
Compare
ad1a31c to
1b6b3bc
Compare
fdf1938 to
3228bf5
Compare
3228bf5 to
aa8e870
Compare
1b6b3bc to
af9ffea
Compare
aa8e870 to
390601e
Compare
af9ffea to
aa10f37
Compare
390601e to
9b07997
Compare
a67d648 to
96a2deb
Compare
policy.generation.dynamo=true on a backend=megatron config is now served end to end: MegatronDynamoGeneration posts native rollouts to the frontend as token-ID completions (dynamo_rollouts), one request per prompt, and generate_async yields the assembled batch. validate_settings replaces the "not served yet" refusal with the checks this mode needs: it refuses NeMo-Gym, VLM, router replay, colocation, and data-plane storage units on the inference cluster, whose placement groups would book the engines' GPUs a second time. Ships an exemplar config, a three-GPU functional test, and the guide section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
Setup runs the generation backend's settings checks before should_use_nemo_gym(), and the Megatron Dynamo checks refuse Gym before the generation block. A Gym config with dynamo on is now told that Gym is not served yet, instead of getting an expose_http_server assertion and then the opposite expose_http_server error. The functional test's ai-dynamo preflight now builds and checks the Megatron Dynamo worker's venv; `uv run --no-sync --extra mcore` ran in the project environment, which does not carry the mcore extra. The test leaves the L1 Dynamo list until the pinned Megatron-LM ships the Dynamo integration, since its own check for that directory would fail every run. LegacyManagedDynamoRuntime no longer warns: the vLLM DynamoGeneration still needs it and has nothing to move to. The guide and the services runtime's docstring say which runtime each backend uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
The guide and the functional test's comment described a decode-then-prefill resume that the fleet no longer performs: every engine resumes in one wave, since persist-mode engines keep their KV across suspend and resume and a handoff that reaches a decode engine still waking is queued. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
…th the code The rollouts test module carried no mcore marker, so no unit-test shard collected it: the Megatron shards keep only mcore-marked tests and the other shards never reach an unmarked test_megatron_*.py under models/generation. It now carries the marker like its siblings. The exemplar's header still described a decode-then-prefill resume; every engine resumes in one wave. generate() kept its "not served yet" docstring and blamed the frontend for the single-sample limit, which is NeMo RL's own contract. The three imports deferred into validate_settings and generate_async are hoisted: the module's top-level chain already loads aiohttp, the data-plane interfaces and router replay, so the deferrals bought nothing and their comments pointed at a cycle that does not exist. The guide's intro and the docs index card said the managed Dynamo integration is vLLM-only; both now name the Megatron prefill/decode path too. The etcd and nats-server lookup is described as the code does it: only NEMO_RL_DYNAMO_BIN_DIR when it is set, otherwise PATH and then the image's Dynamo venv. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
2def71f to
412e7e6
Compare
lauradang
left a comment
There was a problem hiding this comment.
Team review of the Megatron prefill/decode → Dynamo frontend switch-on. The rollout path is cleanly separated and its output contract matches the direct path. Every frontend and Megatron-LM claim below was checked against ai-dynamo v1.5.1 (2093662a) and Megatron-LM ed9c2c8c1c.
Two items for the description, no code change:
truncated: the description says the output matches the direct Megatron path includingtruncated, but the direct path does not set it (megatron_worker.py#L1460-L1465). This path sets it fromfinish_reason == "length", so overlong filtering and the truncation metric differ between direct and prefill/decode runs. Worth one line in the guide too.- Evidence: the GPU path has not run at this head. Before leaving draft, a run of the functional test (on a branch with the Dynamo-enabled Megatron-LM) and
train/token_mult_prob_erroragainst directbackend: megatronon the same model and seed would back the switch-on.
Generated by Claude Code
| # The frontend caps the stop strings of one request. | ||
| _MAX_STOP_STRINGS = 32 |
There was a problem hiding this comment.
nemo_rl/models/generation/megatron/dynamo_rollouts.py:45
1 action item. PR-introduced (copied from the vLLM DynamoGeneration, which has the same cap).
TL;DR — the pinned ai-dynamo 1.5.1 rejects more than 4 stop strings or 4 stop token IDs, so 5–32 pass this check and then fail as an unexplained HTTP 500.
- The frontend's HTTP validator allows 32 (
validate.rs#L446), which is where 32 comes from. - Every completions request then runs
extract_stop_conditions(called frompreprocessor.rs#L2780):if stop.len() > 4 { bail!("stop conditions must be less than 4") }, and the same forstop_token_ids. - That
anyhowerror becomes a generic 500 "Failed to generate completions" (http/service/openai.rs#L653);is_retryableretries it 3×, then raises aRuntimeErrorthat never mentions stop strings.
Upstream fixed this in ai-dynamo/dynamo@9062a667, which is not in the v1.5.1 tag. It is rare with defaults (one stop ID, no stop strings).
AI-1
Cap at what the pinned frontend accepts, check stop_token_ids too (ideally in validate_settings, so it fails before engines start), and update test_merged_stop_strings_enforce_the_frontend_limit to 4/5:
| # The frontend caps the stop strings of one request. | |
| _MAX_STOP_STRINGS = 32 | |
| # ai-dynamo 1.5.1 rejects more than 4 stop strings or stop token IDs per | |
| # request (as an HTTP 500); later releases accept 32. | |
| _MAX_STOP_STRINGS = 4 |
| format_error(response), | ||
| ) | ||
| await asyncio.sleep(_HTTP_RETRY_DELAY_S) | ||
| return parse_completion_response(response, request_url=url) |
There was a problem hiding this comment.
nemo_rl/models/generation/megatron/dynamo_rollouts.py:197
1 action item. PR-introduced (copied from the vLLM DynamoGeneration).
TL;DR — a frontend 503 or refused connection is classed as a DATA failure, so with default budgets a ~5 s frontend outage ends the run.
async_http_post_jsonturns transport errors and 5xx into dicts; after the retriesparse_completion_responseraises a bareRuntimeErrorwith no cause.- That type is outside
_INFRA_TYPES, so_classify_generation_failurewraps it asRolloutDataFailure. - Data failures get
max_data_attempts_per_prompt=2andmax_skipped_prompts=0(config.py): the run stops. As INFRA it would get 5 attempts with 1/2/4/8 s backoff.
Checked on CPU: posting to a closed port gives RuntimeError → FailureClass.DATA → RolloutDataFailure.
AI-1
After the last attempt, raise the infra type for retryable errors; 4xx stays DATA. GenerationUnavailable is what vllm_generation.py raises (a function-local import, as there):
| return parse_completion_response(response, request_url=url) | |
| if is_retryable(response): | |
| from nemo_rl.experience.failures import GenerationUnavailable | |
| raise GenerationUnavailable( | |
| f"Dynamo completion request to {url} failed after " | |
| f"{_HTTP_MAX_ATTEMPTS} attempts: {format_error(response)}" | |
| ) | |
| return parse_completion_response(response, request_url=url) |
| "with policy.generation.dynamo=true; set env.should_use_nemo_gym=" | ||
| "false or leave policy.generation.dynamo false." | ||
| ) | ||
| validate_megatron_dynamo_generation(policy_config["generation"]) |
There was a problem hiding this comment.
nemo_rl/models/generation/megatron/dynamo_generation.py:337
1 action item. PR-introduced.
policy.generation.ignore_eos: true is silently dropped here. The direct path honours it with termination_id=-1 (megatron_worker.py#L1310-L1314), but completion_payload never sends it, so the same config stops at EOS once dynamo: true. Forwarding it is not an exact match: the frontend's apply_ignore_eos also clears stop and stop_token_ids, which the direct path keeps.
AI-1
Refuse it here, with a row in the test_validate_settings table. No shipped config sets it.
| validate_megatron_dynamo_generation(policy_config["generation"]) | |
| validate_megatron_dynamo_generation(policy_config["generation"]) | |
| if policy_config["generation"].get("ignore_eos"): | |
| raise ValueError( | |
| "policy.generation.ignore_eos=true is not supported with " | |
| "policy.generation.dynamo=true: the frontend's ignore_eos also " | |
| "drops stop strings and stop_token_ids." | |
| ) |
| return | ||
| output = await generate_through_frontend( | ||
| self.cfg, | ||
| frontend_url=self.dp_openai_server_base_urls[0] or "", |
There was a problem hiding this comment.
nemo_rl/models/generation/megatron/dynamo_generation.py:1156
1 action item. PR-introduced.
Native rollouts read the frontend URL from dp_openai_server_base_urls, the list setup hands to NeMo-Gym. self._dynamo_frontend_url holds the same URL and is already used here. The next PR in the stack points dp_openai_server_base_urls at the Gym proxies and rewrites this line to _dynamo_frontend_url, so the change can land here. Also, or "" turns a missing URL into an aiohttp InvalidURL, which is retried 3× as a transport error instead of failing at once.
AI-1
| frontend_url=self.dp_openai_server_base_urls[0] or "", | |
| frontend_url=self._dynamo_frontend_url, |
Same idea in generate_through_frontend: stop_strings is a required key, so use cfg["stop_strings"] rather than cfg.get(...).
| raise NotImplementedError( | ||
| "Megatron engines behind Dynamo do not serve rollouts yet; nothing " | ||
| "posts requests to the frontend." | ||
| """Post the batch to the frontend, one request per prompt; yield the assembled batch.""" |
There was a problem hiding this comment.
nemo_rl/models/generation/megatron/dynamo_generation.py:1151
1 action item. The docstring describes a batch, but generate_through_frontend asserts data.size == 1.
| """Post the batch to the frontend, one request per prompt; yield the assembled batch.""" | |
| """Post one single-prompt batch to the frontend; yield its output.""" |
| # resumes them in one wave. The first sync only builds the engines: they | ||
| # start unpaused, so it logs neither a pause nor a resume. | ||
| sync_count=$(grep -Fc "_sync_weights: sync done" "${RUN_LOG}" || true) | ||
| pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true) |
There was a problem hiding this comment.
tests/functional/grpo_dynamo_pd_megatron_single_controller.sh:80
1 action item. PR-introduced.
This check can't fail when the decode engine never suspends. grep -Fc "paused inference engine" matches both prefill's "…paused inference engine scheduling" (megatron_worker.py#L832) and the suspend line "…paused inference engine" (#L848). Prefill alone prints 2 lines per refit, which already meets 2 * (sync_count - 1).
AI-1
Count suspends exactly. Optionally also set export RAY_DEDUP_LOGS=0 near the top, as _sglang_grpo_chaos.py does: Ray strips digits when comparing lines, so the two engines' identical [Rank 0] … lines can be folded into one.
| pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true) | |
| pause_count=$(grep -c "paused inference engine$" "${RUN_LOG}" || true) |
| return decoded | ||
|
|
||
|
|
||
| async def async_http_post_json( |
There was a problem hiding this comment.
nemo_rl/models/generation/megatron/dynamo_rollouts.py:62
1 action item. PR-introduced; design suggestion.
TL;DR — this module copies the vLLM Dynamo client's HTTP, retry and parse code line for line, so the stop-cap and failure-class fixes above have to be made twice, and the Gym PR adds a third copy.
_decode_json_object,async_http_post_jsonandformat_errormatchdynamo/http_client.py#L25-L96.is_retryableandparse_completion_responsematchdynamo/dynamo_generation.py#L57-L130.- The next PR's Gym proxy imports this module's private
_HTTP_MAX_ATTEMPTS/_HTTP_RETRY_DELAY_Sto write another retry loop.
These pieces implement the frontend's /v1/completions contract, which changes with the Dynamo version rather than with the engine, and the Megatron module already imports nemo_rl.models.generation.dynamo.
AI-1
Import the three helpers from http_client.py, and move is_retryable, parse_completion_response and a public retrying post_completion there for both backends and the Gym proxy to share. Keep completion_payload, allowed_new_tokens and single_sample_output here: they read backend-specific config. This also brings in the existing test_dynamo_http_client.py coverage; today every test here monkeypatches async_http_post_json.
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Review
The new rollout path silently ignores the supported EOS override (C1). The functional test also rejects a supported interpreter configuration (C3), and its suspension assertion can pass without prefill suspension (C4).
Additional suggestions cover the duplicated HTTP client (C2), missing quantitative performance evidence (C5), and long in-function comments (C6).
I inspected all 15 changed files and consolidated all three topic reviews. Validation was static only: no tests, linters, GPU runs, or CI queries were executed. The mounted Megatron-Bridge submodule has no materialized upstream source, so I couldn't verify the external Dynamo adapter implementation. The proposed corrections and acceptance checks need verification before adoption.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| "include_stop_str_in_output": True, | ||
| "nvext": {"extra_fields": ["completion_token_ids"]}, | ||
| } | ||
| if cfg["stop_token_ids"] is not None: |
There was a problem hiding this comment.
C1 · 1 action item — please fix in this PR.
TL;DR — [BUG, confidence: 86] The new rollout path silently ignores the request to disable EOS termination.
PR-introduced: completion_payload produces identical requests when the EOS override is true or false. The direct Megatron path honors that override by selecting termination ID -1. With the override enabled and the stop-ID list omitted, normalization fills that list with EOS, which this new branch then sends unchanged. The configured behavior is therefore lost at the request boundary.
AI-1
Action: Reject the unsupported combination in validate_megatron_dynamo_generation, immediately after its backend assertion:
if generation_cfg.get("ignore_eos"):
raise ValueError(
"policy.generation.ignore_eos=true is not supported with "
"policy.generation.dynamo=true"
)This targets a different file, so it isn't a committable suggestion. The external adapter source isn't available here; a complete implementation of EOS-override support can't be justified from the inspected source.
Preservation and proposed acceptance: An otherwise-valid Dynamo config with the override enabled must fail before engine allocation. Absent/false variants must remain accepted, with configured/per-sample stop strings and explicit stop IDs unchanged. Keep the direct-backend regression assertion that the enabled override selects termination ID -1 when Dynamo is disabled. These checks weren't executed.
| return decoded | ||
|
|
||
|
|
||
| async def async_http_post_json( |
There was a problem hiding this comment.
C2 · 1 action item.
TL;DR — [GUIDELINE, confidence: 95] The new module copies the shared Dynamo HTTP client, creating a second place for transport fixes.
PR-introduced maintenance issue, not a demonstrated runtime failure: the decoder, POST helper, and formatter reproduce the existing shared client, including its timeout construction and error contract. The reuse rule calls for using the existing helper.
AI-1
Action: Replace the copied functions with this import in the new module's import block:
from nemo_rl.models.generation.dynamo.http_client import (
async_http_post_json,
format_dynamo_error as format_error,
)Remove the three copied functions and their now-unused JSON/HTTP-library imports. Widen the shared asynchronous helper's signature to:
async def async_http_post_json(
url: str, payload: dict[str, Any], timeout_s: float | None
) -> dict[str, Any]:Keep the Megatron-specific payload, response parser, and retry policy separate. This is a multi-region correction, not a single-line suggestion.
Preservation and proposed acceptance: Retain finite deadlines and no-deadline calls, successful JSON objects, malformed/non-object JSON errors, HTTP error bodies, and transport/timeout errors. Extend the shared client tests to inspect timeout construction for a finite value and None. Retain the rollout payload/output assertions and retry attempt counts. No tests were executed.
| # venv (the mcore extra carries ai-dynamo); etcd and nats-server come from | ||
| # NEMO_RL_DYNAMO_BIN_DIR, PATH, or the image's Dynamo venv. | ||
| MCORE_VENV_ROOT="${NEMO_RL_VENV_DIR:-${PROJECT_ROOT}/venvs}" | ||
| MCORE_PYTHON="${MCORE_VENV_ROOT}/nemo_rl.models.policy.workers.megatron_policy_worker.MegatronDynamoPolicyWorker/bin/python" |
There was a problem hiding this comment.
C3 · 1 action item.
TL;DR — [BUG, confidence: 95] The preflight requires a worker virtual environment even when the engines use the system interpreter.
PR-introduced: with NEMO_RL_PY_EXECUTABLES_SYSTEM=1, the production resolver returns the driver's interpreter, and prefetch deliberately skips that environment. In a dependency-complete checkout without the per-worker virtual environment, the following command still executes the hard-coded path and fails before training. It should check the interpreter the engines will actually use.
AI-1
Action: Replace lines 36–40 with resolution through the actor registry and the existing environment builder:
(cd "${PROJECT_ROOT}" && uv run --no-sync python - <<'PY'
import subprocess
from nemo_rl.distributed.ray_actor_environment_registry import get_actor_python_env
from nemo_rl.utils.venvs import create_local_venv
actor = "nemo_rl.models.policy.workers.megatron_policy_worker.MegatronDynamoPolicyWorker"
python = get_actor_python_env(actor)
if python.startswith("uv"):
python = create_local_venv(python, actor)
subprocess.run(
[python, "-c", 'import importlib.metadata as m; assert m.version("ai-dynamo") == "1.5.1"'],
check=True,
)
PY
)The replacement spans multiple lines, so it isn't an applyable single-line suggestion.
Preservation and proposed acceptance: With the override enabled and all dependencies available, the preflight must check the system interpreter without creating a worker virtual environment. With it disabled, the preflight must build/check the registered worker environment, including a custom virtual-environment root. Both modes must still reject a mismatched Dynamo version. The replacement and checks weren't executed; verify them before adoption.
| # resumes them in one wave. The first sync only builds the engines: they | ||
| # start unpaused, so it logs neither a pause nor a resume. | ||
| sync_count=$(grep -Fc "_sync_weights: sync done" "${RUN_LOG}" || true) | ||
| pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true) |
There was a problem hiding this comment.
C4 · 1 action item.
TL;DR — [TEST, confidence: 95] Scheduling-only pauses satisfy the assertion intended to verify engine suspension.
PR-introduced validation gap: the scheduling pause and actual suspension both match this substring. If the prefill suspension call is omitted, each later refit still contributes one prefill scheduling-pause line and one decode suspension line. For three syncs, that gives four matching lines and passes the four-line threshold. The resume path also logs after a scheduling-only pause, so its count doesn't catch this omission.
AI-1
Action: Exclude the scheduling-only diagnostic from the suspension count:
| pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true) | |
| pause_count=$(grep -F "paused inference engine" "${RUN_LOG}" | grep -Fvc "paused inference engine scheduling" || true) |
Preservation and proposed acceptance: A normal three-step functional run must still pass and retain its scheduling diagnostic. A temporary fault-injection run that omits prefill suspension while retaining its scheduling pause and decode suspension must fail this assertion. The source invariant is that scheduling-only lines contribute zero to the suspension count. These checks weren't executed; verify the suggestion before adoption, without adding a unit test of the shell script.
| generation: | ||
| backend: megatron | ||
| # Serve the engines through a managed Dynamo frontend (prefill/decode only). | ||
| dynamo: true |
There was a problem hiding this comment.
C5 · 1 action item.
TL;DR — [PERF-EVIDENCE, confidence: 95] The feature has no quantitative comparison showing its benefit or training trade-offs.
PR-introduced validation request, not a demonstrated performance regression: the new recipe changes serving and cache behavior, but the supplied description and comments provide no throughput, memory, or convergence measurements. The functional correctness assertions aren't measured results. The selected review policy requires quantitative evidence for new features that affect performance or convergence.
AI-1
Action: Add a matched GPU comparison to the PR description: direct Megatron generation versus Dynamo prefill/decode, using the same model, total GPU budget, workload, seed, and batch settings. Report generated tokens per second or rollout wall time, peak GPU memory, and training/generation log-probability agreement across repeated refits, with commands, commits, and hardware. This is an evidence request, so there is no source patch.
Preservation and acceptance: Keep the linked functional correctness gates. The results should state the measured gain or cost; a smoke run must not stand in for convergence evidence. Any convergence claim needs a matched learning curve. No measurements were executed during this review.
| dp_config = master_config.data_plane | ||
| placement = storage_unit_placement(dp_config) | ||
| if placement is not None and (placement == "all" or "inference" in placement): | ||
| # Planning storage units materializes the named clusters' placement |
There was a problem hiding this comment.
C6 · 1 action item.
TL;DR — [GUIDELINE, confidence: 95] Three new in-function comment blocks exceed the repository's comment-length rule.
PR-introduced maintenance issue, not a correctness failure: the contributing rule excludes comment blocks of three or more lines inside function bodies. These explanations can retain their reasons without interrupting the validation code.
AI-1
Action: Shorten the setup ordering comment, the raw Gym-flag comment, and the storage-placement comment to these replacements, respectively:
# Reject unsupported Gym backends before Gym's server assertions. # Check the raw Gym flag before either set of HTTP-server checks. # Materializing inference would reserve the engines' GPUs twice.This changes three separate regions, so the blocks aren't committable suggestions. Preserve the explanations for validation order, reading the raw flag, and avoiding duplicate GPU reservation. Static acceptance: each linked block is at most two lines, and its surrounding executable code remains unchanged. No patch or formatter check was executed.
PR 4 now carries main's Dynamo rewrite through PR 3. Conflicts: - legacy_managed_runtime.py: deleted, following main (its only user, the old DynamoGeneration fleet, is gone). - managed_runtime.py: docstring from main, which names both the vLLM and Megatron registration paths. - docs/guides/dynamo-generation.md: main's vLLM guide, plus this PR's Megatron prefill/decode section (now on the shared ManagedDynamoRuntime) and pointers to it from the intro and "Not supported". - docs/index.md, L1_Functional_Tests_Dynamo.sh: both sides kept. Signed-off-by: Laura Dang <laurad@nvidia.com>
What this is
PR 5 of 6: turns the feature on. With
policy.generation.backend: megatron+dynamo: true,MegatronDynamoGenerationserves native rollouts through a Dynamo frontend, with separate prefill and decode Megatron engines. PRs 1–4 built, refit and registered the engines but refused the switch; this PR posts the rollouts and replaces the refusal with the checks this mode needs.What it adds
megatron/dynamo_rollouts.py, new).MegatronDynamoGeneration.generate_asyncposts each prompt to the frontend's/v1/completionsas token IDs and asks for the sampled token IDs (nvext.completion_token_ids) and their log-probabilities back, so the frontend tokenizes nothing and the result is the sameGenerationOutputSpecthe direct Megatron path returns (output_ids,logprobs,generation_lengths,unpadded_sequence_lengths,truncated). One request per prompt; the generation budget ismax_new_tokenscapped by the context left after the prompt; configured and per-sample stop strings are merged within the frontend's cap; 408/429/5xx and transport errors are retried a few times.dynamo_cfg.request_timeout_sbounds each request (nullwaits, as the direct path does).generate()raises: the SingleController usesgenerate_asynconly.validate_settingschecks what this path cannot serve instead of refusing the switch: NeMo-Gym through the frontend (follow-up, PR 6), VLM prompts, router replay (the KV handoff does not carry routing-replay buffers), and data-plane storage units placed on theinferencecluster (planning them would book the engines' GPUs a second time). The Gym check reads the rawenv.should_use_nemo_gymflag before the generation checks, so a Gym config is told about Gym rather than aboutexpose_http_server.setup_single_controllernow callsvalidate_settingsbeforeshould_use_nemo_gym(), whose assertions about the HTTP server would otherwise fire first.LegacyManagedDynamoRuntimeno longer warns on construction: the vLLMDynamoGenerationstill needs it and has nothing to move to. The two runtime modules say which backend uses which.examples/configs/grpo_math_1B_dynamo_pd_megatron_single_controller.yaml(one trainer GPU, one prefill engine, one decode engine), the functional testtests/functional/grpo_dynamo_pd_megatron_single_controller.sh(three GPUs; checks that every refit after the first pauses and resumes both engines, that the frontend saw both roles, and that no Dynamo process outlives the run), and a Megatron section indocs/guides/dynamo-generation.md(config boundary, what runs where, per-role layouts, refit sequence, limits). The functional test joinsL1_Functional_Tests_Dynamo.shonce the pinned Megatron-LM shipsmegatron/inference/integrations/dynamo; until then its own preflight would fail every run.Refit keeps the Megatron weight synchronizer's sequence (suspend, prepare weights, transfer, prepare KV cache, resume) from PR 4: prefill stops scheduling first, pins drain, every engine suspends, and every engine resumes in one wave.
Stack
main ← PR 1 (#4801) ← PR 2 (#4912) ← PR 3 (#5023) ← PR 4 (#5024) ← PR 5 (#5025, this) ← PR 6 (#5097)
Review this PR against PR 4. PRs 1–4 change nothing at runtime on their own; PR 5 turns the feature on; PR 6 adds NeMo-Gym serving.
Tests
test_megatron_dynamo_rollouts.py(new): payload shape and sampling fields, greedy pinning, multimodal rejection, context-limit and stop-string caps, response parsing, retry predicate and retry loop.test_megatron_dynamo_generation.py: a rollout through the frontend end to end on the real object, and avalidate_settingstable (accepted; trainer, KV mode, Gym, VLM, colocation, storage-unit placement refusals; Gym refused before the server checks).test_megatron_dynamo_config.py: the old "not served yet" refusal test removed.tests/unit/single_controller/test_setup.py: backend settings are checked before the Gym server assertions.tests/functional/grpo_dynamo_pd_megatron_single_controller.sh(above).Before your PR is "Ready for review"
🤖 Generated with Claude Code