Skip to content

feat(megatron): serve prefill/decode engines through the Dynamo frontend - #5025

Open
lauradang wants to merge 5 commits into
laurad/pd-sc-4-worker-poolfrom
laurad/pd-sc-5-wire-up
Open

lauradang wants to merge 5 commits into
laurad/pd-sc-4-worker-poolfrom
laurad/pd-sc-5-wire-up

Conversation

@lauradang

@lauradang lauradang commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What this is

PR 5 of 6: turns the feature on. With policy.generation.backend: megatron + dynamo: true, MegatronDynamoGeneration serves native rollouts through a Dynamo frontend, with separate prefill and decode Megatron engines. PRs 1–4 built, refit and registered the engines but refused the switch; this PR posts the rollouts and replaces the refusal with the checks this mode needs.

Big picture

What it adds

  • Rollouts through the frontend (megatron/dynamo_rollouts.py, new). MegatronDynamoGeneration.generate_async posts each prompt to the frontend's /v1/completions as token IDs and asks for the sampled token IDs (nvext.completion_token_ids) and their log-probabilities back, so the frontend tokenizes nothing and the result is the same GenerationOutputSpec the direct Megatron path returns (output_ids, logprobs, generation_lengths, unpadded_sequence_lengths, truncated). One request per prompt; the generation budget is max_new_tokens capped by the context left after the prompt; configured and per-sample stop strings are merged within the frontend's cap; 408/429/5xx and transport errors are retried a few times. dynamo_cfg.request_timeout_s bounds each request (null waits, as the direct path does). generate() raises: the SingleController uses generate_async only.
  • validate_settings checks what this path cannot serve instead of refusing the switch: NeMo-Gym through the frontend (follow-up, PR 6), VLM prompts, router replay (the KV handoff does not carry routing-replay buffers), and data-plane storage units placed on the inference cluster (planning them would book the engines' GPUs a second time). The Gym check reads the raw env.should_use_nemo_gym flag before the generation checks, so a Gym config is told about Gym rather than about expose_http_server.
  • Setup runs the backend's settings checks first. setup_single_controller now calls validate_settings before should_use_nemo_gym(), whose assertions about the HTTP server would otherwise fire first.
  • Runtime docstrings. LegacyManagedDynamoRuntime no longer warns on construction: the vLLM DynamoGeneration still needs it and has nothing to move to. The two runtime modules say which backend uses which.
  • Ships the exemplar examples/configs/grpo_math_1B_dynamo_pd_megatron_single_controller.yaml (one trainer GPU, one prefill engine, one decode engine), the functional test tests/functional/grpo_dynamo_pd_megatron_single_controller.sh (three GPUs; checks that every refit after the first pauses and resumes both engines, that the frontend saw both roles, and that no Dynamo process outlives the run), and a Megatron section in docs/guides/dynamo-generation.md (config boundary, what runs where, per-role layouts, refit sequence, limits). The functional test joins L1_Functional_Tests_Dynamo.sh once the pinned Megatron-LM ships megatron/inference/integrations/dynamo; until then its own preflight would fail every run.

First refit: weights in, engines built, endpoints registered, rollouts allowed

Refit keeps the Megatron weight synchronizer's sequence (suspend, prepare weights, transfer, prepare KV cache, resume) from PR 4: prefill stops scheduling first, pins drain, every engine suspends, and every engine resumes in one wave.

Stack

main ← PR 1 (#4801) ← PR 2 (#4912) ← PR 3 (#5023) ← PR 4 (#5024) ← PR 5 (#5025, this) ← PR 6 (#5097)

Review this PR against PR 4. PRs 1–4 change nothing at runtime on their own; PR 5 turns the feature on; PR 6 adds NeMo-Gym serving.

Tests

  • test_megatron_dynamo_rollouts.py (new): payload shape and sampling fields, greedy pinning, multimodal rejection, context-limit and stop-string caps, response parsing, retry predicate and retry loop.
  • test_megatron_dynamo_generation.py: a rollout through the frontend end to end on the real object, and a validate_settings table (accepted; trainer, KV mode, Gym, VLM, colocation, storage-unit placement refusals; Gym refused before the server checks).
  • test_megatron_dynamo_config.py: the old "not served yet" refusal test removed.
  • tests/unit/single_controller/test_setup.py: backend settings are checked before the Gym server assertions.
  • Functional: tests/functional/grpo_dynamo_pd_megatron_single_controller.sh (above).

Before your PR is "Ready for review"

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? (CPU unit suites only; GPU path not yet run on this base)
  • Did you add or update any necessary documentation?

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 2, 2026
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 305608c to f3fd7e2 Compare November 2, 2026 17:50
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch 2 times, most recently from a265e75 to eb743cb Compare November 4, 2026 03:00
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from f3fd7e2 to 37213bc Compare November 4, 2026 03:00
@lauradang lauradang changed the title feat(dynamo): run Megatron prefill/decode workers behind the managed frontend feat(megatron): serve prefill/decode engines through the Dynamo frontend Nov 4, 2026
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 37213bc to 74c9462 Compare November 4, 2026 03:39
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch 2 times, most recently from 19bd47b to 5dae5e8 Compare November 4, 2026 05:36
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch 2 times, most recently from d4290d4 to fdf1938 Compare November 5, 2026 09:26
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch 2 times, most recently from ad1a31c to 1b6b3bc Compare November 5, 2026 09:27
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from fdf1938 to 3228bf5 Compare November 5, 2026 09:27
@terrykong terrykong added the CI:L1 Run doctests, unit tests, and functional tests label Nov 5, 2026
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 3228bf5 to aa8e870 Compare November 5, 2026 21:41
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch from 1b6b3bc to af9ffea Compare November 5, 2026 21:41
@terrykong terrykong added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Nov 5, 2026
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from aa8e870 to 390601e Compare November 5, 2026 23:59
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch from af9ffea to aa10f37 Compare November 5, 2026 23:59
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 390601e to 9b07997 Compare November 6, 2026 00:05
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch 2 times, most recently from a67d648 to 96a2deb Compare November 6, 2026 22:38
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

lauradang and others added 4 commits November 6, 2026 21:02
policy.generation.dynamo=true on a backend=megatron config is now served
end to end: MegatronDynamoGeneration posts native rollouts to the frontend as
token-ID completions (dynamo_rollouts), one request per prompt, and
generate_async yields the assembled batch. validate_settings replaces the
"not served yet" refusal with the checks this mode needs: it refuses
NeMo-Gym, VLM, router replay, colocation, and data-plane storage units on
the inference cluster, whose placement groups would book the engines' GPUs
a second time.

Ships an exemplar config, a three-GPU functional test, and the guide section.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
Setup runs the generation backend's settings checks before
should_use_nemo_gym(), and the Megatron Dynamo checks refuse Gym before the
generation block. A Gym config with dynamo on is now told that Gym is not
served yet, instead of getting an expose_http_server assertion and then the
opposite expose_http_server error.

The functional test's ai-dynamo preflight now builds and checks the Megatron
Dynamo worker's venv; `uv run --no-sync --extra mcore` ran in the project
environment, which does not carry the mcore extra. The test leaves the L1
Dynamo list until the pinned Megatron-LM ships the Dynamo integration, since
its own check for that directory would fail every run.

LegacyManagedDynamoRuntime no longer warns: the vLLM DynamoGeneration still
needs it and has nothing to move to. The guide and the services runtime's
docstring say which runtime each backend uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
The guide and the functional test's comment described a decode-then-prefill
resume that the fleet no longer performs: every engine resumes in one wave,
since persist-mode engines keep their KV across suspend and resume and a
handoff that reaches a decode engine still waking is queued.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
…th the code

The rollouts test module carried no mcore marker, so no unit-test shard
collected it: the Megatron shards keep only mcore-marked tests and the other
shards never reach an unmarked test_megatron_*.py under models/generation. It
now carries the marker like its siblings.

The exemplar's header still described a decode-then-prefill resume; every
engine resumes in one wave. generate() kept its "not served yet" docstring
and blamed the frontend for the single-sample limit, which is NeMo RL's own
contract. The three imports deferred into validate_settings and
generate_async are hoisted: the module's top-level chain already loads
aiohttp, the data-plane interfaces and router replay, so the deferrals bought
nothing and their comments pointed at a cycle that does not exist.

The guide's intro and the docs index card said the managed Dynamo integration
is vLLM-only; both now name the Megatron prefill/decode path too. The etcd and
nats-server lookup is described as the code does it: only NEMO_RL_DYNAMO_BIN_DIR
when it is set, otherwise PATH and then the image's Dynamo venv.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
@lauradang
lauradang force-pushed the laurad/pd-sc-5-wire-up branch from 2def71f to 412e7e6 Compare November 7, 2026 05:12
@lauradang
lauradang marked this pull request as ready for review November 10, 2026 20:05

@lauradang lauradang left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Team review of the Megatron prefill/decode → Dynamo frontend switch-on. The rollout path is cleanly separated and its output contract matches the direct path. Every frontend and Megatron-LM claim below was checked against ai-dynamo v1.5.1 (2093662a) and Megatron-LM ed9c2c8c1c.

Two items for the description, no code change:

  • truncated: the description says the output matches the direct Megatron path including truncated, but the direct path does not set it (megatron_worker.py#L1460-L1465). This path sets it from finish_reason == "length", so overlong filtering and the truncation metric differ between direct and prefill/decode runs. Worth one line in the guide too.
  • Evidence: the GPU path has not run at this head. Before leaving draft, a run of the functional test (on a branch with the Dynamo-enabled Megatron-LM) and train/token_mult_prob_error against direct backend: megatron on the same model and seed would back the switch-on.

Generated by Claude Code

Comment on lines +44 to +45
# The frontend caps the stop strings of one request.
_MAX_STOP_STRINGS = 32

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/megatron/dynamo_rollouts.py:45

1 action item. PR-introduced (copied from the vLLM DynamoGeneration, which has the same cap).

TL;DR — the pinned ai-dynamo 1.5.1 rejects more than 4 stop strings or 4 stop token IDs, so 5–32 pass this check and then fail as an unexplained HTTP 500.

  1. The frontend's HTTP validator allows 32 (validate.rs#L446), which is where 32 comes from.
  2. Every completions request then runs extract_stop_conditions (called from preprocessor.rs#L2780): if stop.len() > 4 { bail!("stop conditions must be less than 4") }, and the same for stop_token_ids.
  3. That anyhow error becomes a generic 500 "Failed to generate completions" (http/service/openai.rs#L653); is_retryable retries it 3×, then raises a RuntimeError that never mentions stop strings.

Upstream fixed this in ai-dynamo/dynamo@9062a667, which is not in the v1.5.1 tag. It is rare with defaults (one stop ID, no stop strings).

AI-1

Cap at what the pinned frontend accepts, check stop_token_ids too (ideally in validate_settings, so it fails before engines start), and update test_merged_stop_strings_enforce_the_frontend_limit to 4/5:

Suggested change
# The frontend caps the stop strings of one request.
_MAX_STOP_STRINGS = 32
# ai-dynamo 1.5.1 rejects more than 4 stop strings or stop token IDs per
# request (as an HTTP 500); later releases accept 32.
_MAX_STOP_STRINGS = 4

format_error(response),
)
await asyncio.sleep(_HTTP_RETRY_DELAY_S)
return parse_completion_response(response, request_url=url)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/megatron/dynamo_rollouts.py:197

1 action item. PR-introduced (copied from the vLLM DynamoGeneration).

TL;DR — a frontend 503 or refused connection is classed as a DATA failure, so with default budgets a ~5 s frontend outage ends the run.

  1. async_http_post_json turns transport errors and 5xx into dicts; after the retries parse_completion_response raises a bare RuntimeError with no cause.
  2. That type is outside _INFRA_TYPES, so _classify_generation_failure wraps it as RolloutDataFailure.
  3. Data failures get max_data_attempts_per_prompt=2 and max_skipped_prompts=0 (config.py): the run stops. As INFRA it would get 5 attempts with 1/2/4/8 s backoff.

Checked on CPU: posting to a closed port gives RuntimeError → FailureClass.DATA → RolloutDataFailure.

AI-1

After the last attempt, raise the infra type for retryable errors; 4xx stays DATA. GenerationUnavailable is what vllm_generation.py raises (a function-local import, as there):

Suggested change
return parse_completion_response(response, request_url=url)
if is_retryable(response):
from nemo_rl.experience.failures import GenerationUnavailable
raise GenerationUnavailable(
f"Dynamo completion request to {url} failed after "
f"{_HTTP_MAX_ATTEMPTS} attempts: {format_error(response)}"
)
return parse_completion_response(response, request_url=url)

"with policy.generation.dynamo=true; set env.should_use_nemo_gym="
"false or leave policy.generation.dynamo false."
)
validate_megatron_dynamo_generation(policy_config["generation"])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/megatron/dynamo_generation.py:337

1 action item. PR-introduced.

policy.generation.ignore_eos: true is silently dropped here. The direct path honours it with termination_id=-1 (megatron_worker.py#L1310-L1314), but completion_payload never sends it, so the same config stops at EOS once dynamo: true. Forwarding it is not an exact match: the frontend's apply_ignore_eos also clears stop and stop_token_ids, which the direct path keeps.

AI-1

Refuse it here, with a row in the test_validate_settings table. No shipped config sets it.

Suggested change
validate_megatron_dynamo_generation(policy_config["generation"])
validate_megatron_dynamo_generation(policy_config["generation"])
if policy_config["generation"].get("ignore_eos"):
raise ValueError(
"policy.generation.ignore_eos=true is not supported with "
"policy.generation.dynamo=true: the frontend's ignore_eos also "
"drops stop strings and stop_token_ids."
)

return
output = await generate_through_frontend(
self.cfg,
frontend_url=self.dp_openai_server_base_urls[0] or "",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/megatron/dynamo_generation.py:1156

1 action item. PR-introduced.

Native rollouts read the frontend URL from dp_openai_server_base_urls, the list setup hands to NeMo-Gym. self._dynamo_frontend_url holds the same URL and is already used here. The next PR in the stack points dp_openai_server_base_urls at the Gym proxies and rewrites this line to _dynamo_frontend_url, so the change can land here. Also, or "" turns a missing URL into an aiohttp InvalidURL, which is retried 3× as a transport error instead of failing at once.

AI-1

Suggested change
frontend_url=self.dp_openai_server_base_urls[0] or "",
frontend_url=self._dynamo_frontend_url,

Same idea in generate_through_frontend: stop_strings is a required key, so use cfg["stop_strings"] rather than cfg.get(...).

raise NotImplementedError(
"Megatron engines behind Dynamo do not serve rollouts yet; nothing "
"posts requests to the frontend."
"""Post the batch to the frontend, one request per prompt; yield the assembled batch."""

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/megatron/dynamo_generation.py:1151

1 action item. The docstring describes a batch, but generate_through_frontend asserts data.size == 1.

Suggested change
"""Post the batch to the frontend, one request per prompt; yield the assembled batch."""
"""Post one single-prompt batch to the frontend; yield its output."""

# resumes them in one wave. The first sync only builds the engines: they
# start unpaused, so it logs neither a pause nor a resume.
sync_count=$(grep -Fc "_sync_weights: sync done" "${RUN_LOG}" || true)
pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/functional/grpo_dynamo_pd_megatron_single_controller.sh:80

1 action item. PR-introduced.

This check can't fail when the decode engine never suspends. grep -Fc "paused inference engine" matches both prefill's "…paused inference engine scheduling" (megatron_worker.py#L832) and the suspend line "…paused inference engine" (#L848). Prefill alone prints 2 lines per refit, which already meets 2 * (sync_count - 1).

AI-1

Count suspends exactly. Optionally also set export RAY_DEDUP_LOGS=0 near the top, as _sglang_grpo_chaos.py does: Ray strips digits when comparing lines, so the two engines' identical [Rank 0] … lines can be folded into one.

Suggested change
pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true)
pause_count=$(grep -c "paused inference engine$" "${RUN_LOG}" || true)

return decoded


async def async_http_post_json(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/megatron/dynamo_rollouts.py:62

1 action item. PR-introduced; design suggestion.

TL;DR — this module copies the vLLM Dynamo client's HTTP, retry and parse code line for line, so the stop-cap and failure-class fixes above have to be made twice, and the Gym PR adds a third copy.

These pieces implement the frontend's /v1/completions contract, which changes with the Dynamo version rather than with the engine, and the Megatron module already imports nemo_rl.models.generation.dynamo.

AI-1

Import the three helpers from http_client.py, and move is_retryable, parse_completion_response and a public retrying post_completion there for both backends and the Gym proxy to share. Keep completion_payload, allowed_new_tokens and single_sample_output here: they read backend-specific config. This also brings in the existing test_dynamo_http_client.py coverage; today every test here monkeypatches async_http_post_json.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

The new rollout path silently ignores the supported EOS override (C1). The functional test also rejects a supported interpreter configuration (C3), and its suspension assertion can pass without prefill suspension (C4).

Additional suggestions cover the duplicated HTTP client (C2), missing quantitative performance evidence (C5), and long in-function comments (C6).

I inspected all 15 changed files and consolidated all three topic reviews. Validation was static only: no tests, linters, GPU runs, or CI queries were executed. The mounted Megatron-Bridge submodule has no materialized upstream source, so I couldn't verify the external Dynamo adapter implementation. The proposed corrections and acceptance checks need verification before adoption.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

"include_stop_str_in_output": True,
"nvext": {"extra_fields": ["completion_token_ids"]},
}
if cfg["stop_token_ids"] is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C1 · 1 action item — please fix in this PR.
TL;DR — [BUG, confidence: 86] The new rollout path silently ignores the request to disable EOS termination.

PR-introduced: completion_payload produces identical requests when the EOS override is true or false. The direct Megatron path honors that override by selecting termination ID -1. With the override enabled and the stop-ID list omitted, normalization fills that list with EOS, which this new branch then sends unchanged. The configured behavior is therefore lost at the request boundary.

AI-1

Action: Reject the unsupported combination in validate_megatron_dynamo_generation, immediately after its backend assertion:

if generation_cfg.get("ignore_eos"):
    raise ValueError(
        "policy.generation.ignore_eos=true is not supported with "
        "policy.generation.dynamo=true"
    )

This targets a different file, so it isn't a committable suggestion. The external adapter source isn't available here; a complete implementation of EOS-override support can't be justified from the inspected source.

Preservation and proposed acceptance: An otherwise-valid Dynamo config with the override enabled must fail before engine allocation. Absent/false variants must remain accepted, with configured/per-sample stop strings and explicit stop IDs unchanged. Keep the direct-backend regression assertion that the enabled override selects termination ID -1 when Dynamo is disabled. These checks weren't executed.

return decoded


async def async_http_post_json(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C2 · 1 action item.
TL;DR — [GUIDELINE, confidence: 95] The new module copies the shared Dynamo HTTP client, creating a second place for transport fixes.

PR-introduced maintenance issue, not a demonstrated runtime failure: the decoder, POST helper, and formatter reproduce the existing shared client, including its timeout construction and error contract. The reuse rule calls for using the existing helper.

AI-1

Action: Replace the copied functions with this import in the new module's import block:

from nemo_rl.models.generation.dynamo.http_client import (
    async_http_post_json,
    format_dynamo_error as format_error,
)

Remove the three copied functions and their now-unused JSON/HTTP-library imports. Widen the shared asynchronous helper's signature to:

async def async_http_post_json(
    url: str, payload: dict[str, Any], timeout_s: float | None
) -> dict[str, Any]:

Keep the Megatron-specific payload, response parser, and retry policy separate. This is a multi-region correction, not a single-line suggestion.

Preservation and proposed acceptance: Retain finite deadlines and no-deadline calls, successful JSON objects, malformed/non-object JSON errors, HTTP error bodies, and transport/timeout errors. Extend the shared client tests to inspect timeout construction for a finite value and None. Retain the rollout payload/output assertions and retry attempt counts. No tests were executed.

# venv (the mcore extra carries ai-dynamo); etcd and nats-server come from
# NEMO_RL_DYNAMO_BIN_DIR, PATH, or the image's Dynamo venv.
MCORE_VENV_ROOT="${NEMO_RL_VENV_DIR:-${PROJECT_ROOT}/venvs}"
MCORE_PYTHON="${MCORE_VENV_ROOT}/nemo_rl.models.policy.workers.megatron_policy_worker.MegatronDynamoPolicyWorker/bin/python"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C3 · 1 action item.
TL;DR — [BUG, confidence: 95] The preflight requires a worker virtual environment even when the engines use the system interpreter.

PR-introduced: with NEMO_RL_PY_EXECUTABLES_SYSTEM=1, the production resolver returns the driver's interpreter, and prefetch deliberately skips that environment. In a dependency-complete checkout without the per-worker virtual environment, the following command still executes the hard-coded path and fails before training. It should check the interpreter the engines will actually use.

AI-1

Action: Replace lines 36–40 with resolution through the actor registry and the existing environment builder:

(cd "${PROJECT_ROOT}" && uv run --no-sync python - <<'PY'
import subprocess

from nemo_rl.distributed.ray_actor_environment_registry import get_actor_python_env
from nemo_rl.utils.venvs import create_local_venv

actor = "nemo_rl.models.policy.workers.megatron_policy_worker.MegatronDynamoPolicyWorker"
python = get_actor_python_env(actor)
if python.startswith("uv"):
    python = create_local_venv(python, actor)
subprocess.run(
    [python, "-c", 'import importlib.metadata as m; assert m.version("ai-dynamo") == "1.5.1"'],
    check=True,
)
PY
)

The replacement spans multiple lines, so it isn't an applyable single-line suggestion.

Preservation and proposed acceptance: With the override enabled and all dependencies available, the preflight must check the system interpreter without creating a worker virtual environment. With it disabled, the preflight must build/check the registered worker environment, including a custom virtual-environment root. Both modes must still reject a mismatched Dynamo version. The replacement and checks weren't executed; verify them before adoption.

# resumes them in one wave. The first sync only builds the engines: they
# start unpaused, so it logs neither a pause nor a resume.
sync_count=$(grep -Fc "_sync_weights: sync done" "${RUN_LOG}" || true)
pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C4 · 1 action item.
TL;DR — [TEST, confidence: 95] Scheduling-only pauses satisfy the assertion intended to verify engine suspension.

PR-introduced validation gap: the scheduling pause and actual suspension both match this substring. If the prefill suspension call is omitted, each later refit still contributes one prefill scheduling-pause line and one decode suspension line. For three syncs, that gives four matching lines and passes the four-line threshold. The resume path also logs after a scheduling-only pause, so its count doesn't catch this omission.

AI-1

Action: Exclude the scheduling-only diagnostic from the suspension count:

Suggested change
pause_count=$(grep -Fc "paused inference engine" "${RUN_LOG}" || true)
pause_count=$(grep -F "paused inference engine" "${RUN_LOG}" | grep -Fvc "paused inference engine scheduling" || true)

Preservation and proposed acceptance: A normal three-step functional run must still pass and retain its scheduling diagnostic. A temporary fault-injection run that omits prefill suspension while retaining its scheduling pause and decode suspension must fail this assertion. The source invariant is that scheduling-only lines contribute zero to the suspension count. These checks weren't executed; verify the suggestion before adoption, without adding a unit test of the shell script.

generation:
backend: megatron
# Serve the engines through a managed Dynamo frontend (prefill/decode only).
dynamo: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C5 · 1 action item.
TL;DR — [PERF-EVIDENCE, confidence: 95] The feature has no quantitative comparison showing its benefit or training trade-offs.

PR-introduced validation request, not a demonstrated performance regression: the new recipe changes serving and cache behavior, but the supplied description and comments provide no throughput, memory, or convergence measurements. The functional correctness assertions aren't measured results. The selected review policy requires quantitative evidence for new features that affect performance or convergence.

AI-1

Action: Add a matched GPU comparison to the PR description: direct Megatron generation versus Dynamo prefill/decode, using the same model, total GPU budget, workload, seed, and batch settings. Report generated tokens per second or rollout wall time, peak GPU memory, and training/generation log-probability agreement across repeated refits, with commands, commits, and hardware. This is an evidence request, so there is no source patch.

Preservation and acceptance: Keep the linked functional correctness gates. The results should state the measured gain or cost; a smoke run must not stand in for convergence evidence. Any convergence claim needs a matched learning curve. No measurements were executed during this review.

dp_config = master_config.data_plane
placement = storage_unit_placement(dp_config)
if placement is not None and (placement == "all" or "inference" in placement):
# Planning storage units materializes the named clusters' placement

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C6 · 1 action item.
TL;DR — [GUIDELINE, confidence: 95] Three new in-function comment blocks exceed the repository's comment-length rule.

PR-introduced maintenance issue, not a correctness failure: the contributing rule excludes comment blocks of three or more lines inside function bodies. These explanations can retain their reasons without interrupting the validation code.

AI-1

Action: Shorten the setup ordering comment, the raw Gym-flag comment, and the storage-placement comment to these replacements, respectively:

    # Reject unsupported Gym backends before Gym's server assertions.
        # Check the raw Gym flag before either set of HTTP-server checks.
            # Materializing inference would reserve the engines' GPUs twice.

This changes three separate regions, so the blocks aren't committable suggestions. Preserve the explanations for validation order, reading the raw flag, and avoiding duplicate GPU reservation. Static acceptance: each linked block is at most two lines, and its surrounding executable code remains unchanged. No patch or formatter check was executed.

PR 4 now carries main's Dynamo rewrite through PR 3. Conflicts:
- legacy_managed_runtime.py: deleted, following main (its only user,
  the old DynamoGeneration fleet, is gone).
- managed_runtime.py: docstring from main, which names both the vLLM
  and Megatron registration paths.
- docs/guides/dynamo-generation.md: main's vLLM guide, plus this PR's
  Megatron prefill/decode section (now on the shared ManagedDynamoRuntime)
  and pointers to it from the intro and "Not supported".
- docs/index.md, L1_Functional_Tests_Dynamo.sh: both sides kept.

Signed-off-by: Laura Dang <laurad@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants