Repository navigation
Conversation
Six call sites in grpo, ppo, distillation and the single-controller setup
propagated `policy.hf_config_overrides` to the generation engine by wholesale
assignment:
vllm_kwargs["hf_overrides"] = policy_config.get("hf_config_overrides", {})
Anything the user set under `policy.generation.vllm_kwargs.hf_overrides` was
discarded silently, even when the policy declared no overrides of its own.
Generation-only overrides are needed when the rollout engine's view of the HF
config must be corrected without changing the trainer (e.g. RoPE scaling for
a trust_remote_code model); dropping them made generation and trainer logprobs
disagree with nothing in the config or logs to show why.
Add `merge_policy_hf_overrides` next to the other vLLM config normalizers and
route all six sites through it. Policy overrides still win on conflicting keys
(the trainer/generation RoPE invariant is unchanged), conflicts emit a warning
naming the keys, and generation-only keys survive. This is the same
overwrite-vs-merge bug `_merge_fp8_kwargs` already fixes one layer down
(NVIDIA-NeMo#1413 / NVIDIA-NeMo#2904).
Closes NVIDIA-NeMo#3728
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
|
/ok to test 9f6be7b |
nvegesna-netizen
left a comment
There was a problem hiding this comment.
Thanks for picking up #3728 and for covering all six setup sites. I had opened #4949 for the same bug without noticing this PR, which came first, so I'm deferring to this one. One idea from #4949 seems worth bringing over (C1): raise on a conflicting key instead of warning.
Notes from the #4949 review that also hold here:
- Every setup site that builds a vLLM or Dynamo engine from a training policy goes through the new helper; none is missed.
- Comparing whole values with
!=is right: vLLM replaces dict-valued keys such asrope_scalingwhole (_apply_dict_overrides), so a recursive merge would hand vLLM a dict the trainer never sees. - Rebase note: open PR #4228 merges these two dicts inline at its grpo vLLM site (grpo.py L1543-L1548), letting the vLLM value win. Its recipe's keys don't overlap the policy's, so whichever PR lands second can take this helper without changing that recipe.
Generated by Claude Code
| if conflicts: | ||
| warnings.warn( | ||
| "policy.hf_config_overrides overrides policy.generation.vllm_kwargs." | ||
| f"hf_overrides for {conflicts}; the training policy's value is used so " | ||
| "the generation engine stays in sync with the trainer.", | ||
| stacklevel=2, | ||
| ) |
There was a problem hiding this comment.
C1 · 1 action item.
TL;DR — when both override dicts set one key to different values, vLLM gets the policy value and the run goes on with only a warning in the driver log. Elsewhere the repo raises a setup-time ValueError when two user settings disagree.
For example, policy.hf_config_overrides.rope_theta: 1e6 plus policy.generation.vllm_kwargs.hf_overrides.rope_theta: 5e5: the run starts and vLLM uses 1e6. The user asked for two different values and is never asked to pick one.
Other places in the repo reject the config instead of picking a winner:
_reject_conflicting_engine_kwargs: "nvfp4_pertoken cannot overwrite explicit vLLM settings".- Dynamo's argv builder raises "is set by both" (arguments.py L69-L73).
No recipe on any public branch sets one key to different values in these two dicts, so a raise blocks no existing config.
AI-1
Could you replace the warning with a ValueError that names the keys and the fix?
if conflicts:
raise ValueError(
f"policy.generation.vllm_kwargs.hf_overrides sets {conflicts} to values "
"that differ from policy.hf_config_overrides. The trainer and vLLM must "
"load the same HF config; set these keys only in policy.hf_config_overrides."
)This is a plain block, not a one-click suggestion, because three other spots describe the warning and need to change with it:
- the docstring sentence "on a key collision the policy value wins and a warning names the key" (config.py L50-L51);
test_policy_wins_on_conflict_and_warnsbecomespytest.raises(ValueError, match=r"\['rope_theta'\]")and drops its result assert;- the guide's "on a conflict the policy value wins with a warning" (yarn-long-context.md L65).
Where the new error fires in distillation and single-controller setup (no action needed)
With a raise, where the call sits starts to matter:
- In distillation, the merge runs after the teacher policy has loaded.
- In non-colocated single-controller runs, it runs inside
_build_generation, in parallel with the trainer build.
Moving these two calls earlier, before any model is built (distillation already has an early vLLM block), would make the error appear first. Not part of this ask.
There was a problem hiding this comment.
Done in ff378e1: conflicting keys now raise your ValueError (names the keys and says to set them only in policy.hf_config_overrides). Updated the docstring, replaced the warn test with test_conflicting_values_raise (pytest.raises(ValueError, match=r"\['rope_theta'\]")), and changed the YaRN guide sentence. Equal values and generation-only keys behave as before. I also checked that no config under examples/configs or research/ sets vllm_kwargs.hf_overrides, so nothing existing trips it. Left the call-site ordering in distillation / non-colocated SC as is, per your note. Thanks for the careful review and for deferring #4949.
Review on NVIDIA-NeMo#4907 (C1): when policy.hf_config_overrides and policy.generation.vllm_kwargs.hf_overrides set one key to different values, the run started with only a driver-log warning and vLLM silently used the policy value. Elsewhere the repo rejects disagreeing user settings at setup (nvfp4_pertoken engine kwargs, Dynamo argv builder). merge_policy_hf_overrides now raises a ValueError naming the keys and the fix. Equal values and generation-only keys are unchanged. The docstring, the conflict test and the YaRN guide are updated to match. No shipped config sets vllm_kwargs.hf_overrides, so no existing recipe is affected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
|
/ok to test ff378e1 |
NVIDIA-NeMo#4905 removed the separate dynamo generation backend, so the dynamo branches this PR patched and their test are gone; dynamo now runs on the vLLM path, which already merges hf_config_overrides. Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@yuki-97 @RayenTian merged main. #4905 removed the separate dynamo backend, so this PR's dynamo sites are gone (description updated); the vLLM merge still covers dynamo. Needs a |
What does this PR do?
Fixes #3728. Six call sites propagated
policy.hf_config_overridesto the generation engine by wholesale assignment:so anything the user set under
policy.generation.vllm_kwargs.hf_overrideswas thrown away, even when the policy declared no overrides of its own, with nothing logged. Generation-only overrides are legitimately needed when the rollout engine's view of the HF config has to be corrected without touching the trainer (the issue's RoPE-scaling case), and dropping them made generation and trainer logprobs disagree with no signal at launch.This is the same overwrite-vs-merge bug that
_merge_fp8_kwargsalready fixes one layer down in the worker (#1413, reverted in #2188, re-fixed in #2904); the algorithm entry points were never updated.Changes
nemo_rl/models/generation/vllm/config.py: newmerge_policy_hf_overrides(vllm_kwargs, policy_hf_config_overrides), placed with the other vLLM config normalizers. Semantics: generation-only keys survive, the policy wins on conflicting keys so the trainer/generation RoPE invariant validated inmegatron/setup.pyis unchanged, and a conflict emits a warning naming the keys instead of resolving silently.grpo.py,ppo.py,distillation.py,single_controller_utils/setup.py. (The two dynamo sites this PR first patched were removed by feat(dynamo)!: serve NeMo RL's own vLLM rollouts through Dynamo's backend SDK #4905; dynamo now runs on the vLLM path, so the vLLM sites cover it.)tests/unit/models/generation/test_vllm_policy_hf_overrides.py: pins the merge behavior (survival, precedence, warning, no-warning on equal values, empty cases, no aliasing, non-mapping rejected).docs/guides/yarn-long-context.md: one sentence on the merge semantics.Testing
ruff check,ruff check --select I,ruff format --checkclean.🤖 Generated with Claude Code