Skip to content

fix(vllm): merge policy hf_config_overrides into generation hf_overrides - #4907

Open
lonexreb wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
lonexreb:fix/3728-merge-hf-overrides
Open

lonexreb wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
lonexreb:fix/3728-merge-hf-overrides

Conversation

@lonexreb

@lonexreb lonexreb commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #3728. Six call sites propagated policy.hf_config_overrides to the generation engine by wholesale assignment:

vllm_kwargs["hf_overrides"] = policy_config.get("hf_config_overrides", {})

so anything the user set under policy.generation.vllm_kwargs.hf_overrides was thrown away, even when the policy declared no overrides of its own, with nothing logged. Generation-only overrides are legitimately needed when the rollout engine's view of the HF config has to be corrected without touching the trainer (the issue's RoPE-scaling case), and dropping them made generation and trainer logprobs disagree with no signal at launch.

This is the same overwrite-vs-merge bug that _merge_fp8_kwargs already fixes one layer down in the worker (#1413, reverted in #2188, re-fixed in #2904); the algorithm entry points were never updated.

Changes

  • nemo_rl/models/generation/vllm/config.py: new merge_policy_hf_overrides(vllm_kwargs, policy_hf_config_overrides), placed with the other vLLM config normalizers. Semantics: generation-only keys survive, the policy wins on conflicting keys so the trainer/generation RoPE invariant validated in megatron/setup.py is unchanged, and a conflict emits a warning naming the keys instead of resolving silently.
  • Routed all four sites through it: grpo.py, ppo.py, distillation.py, single_controller_utils/setup.py. (The two dynamo sites this PR first patched were removed by feat(dynamo)!: serve NeMo RL's own vLLM rollouts through Dynamo's backend SDK #4905; dynamo now runs on the vLLM path, so the vLLM sites cover it.)
  • tests/unit/models/generation/test_vllm_policy_hf_overrides.py: pins the merge behavior (survival, precedence, warning, no-warning on equal values, empty cases, no aliasing, non-mapping rejected).
  • docs/guides/yarn-long-context.md: one sentence on the merge semantics.

Testing

tests/unit/models/generation/test_vllm_policy_hf_overrides.py   9 passed
tests/unit/models/generation/test_vllm_fp8_hf_overrides.py      9 passed
test_grpo / test_ppo / test_distillation / single_controller test_setup
  -k "hf_overrides or vllm_kwargs or dynamo or generation_config"  21 passed

ruff check, ruff check --select I, ruff format --check clean.

🤖 Generated with Claude Code

Six call sites in grpo, ppo, distillation and the single-controller setup
propagated `policy.hf_config_overrides` to the generation engine by wholesale
assignment:

    vllm_kwargs["hf_overrides"] = policy_config.get("hf_config_overrides", {})

Anything the user set under `policy.generation.vllm_kwargs.hf_overrides` was
discarded silently, even when the policy declared no overrides of its own.
Generation-only overrides are needed when the rollout engine's view of the HF
config must be corrected without changing the trainer (e.g. RoPE scaling for
a trust_remote_code model); dropping them made generation and trainer logprobs
disagree with nothing in the config or logs to show why.

Add `merge_policy_hf_overrides` next to the other vLLM config normalizers and
route all six sites through it. Policy overrides still win on conflicting keys
(the trainer/generation RoPE invariant is unchanged), conflicts emit a warning
naming the keys, and generation-only keys survive. This is the same
overwrite-vs-merge bug `_merge_fp8_kwargs` already fixes one layer down
(NVIDIA-NeMo#1413 / NVIDIA-NeMo#2904).

Closes NVIDIA-NeMo#3728

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
@lonexreb

Copy link
Copy Markdown
Contributor Author

/ok to test 9f6be7b

@copy-pr-bot

copy-pr-bot Bot commented Oct 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 25, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 27, 2026

@nvegesna-netizen nvegesna-netizen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up #3728 and for covering all six setup sites. I had opened #4949 for the same bug without noticing this PR, which came first, so I'm deferring to this one. One idea from #4949 seems worth bringing over (C1): raise on a conflicting key instead of warning.

Notes from the #4949 review that also hold here:

  • Every setup site that builds a vLLM or Dynamo engine from a training policy goes through the new helper; none is missed.
  • Comparing whole values with != is right: vLLM replaces dict-valued keys such as rope_scaling whole (_apply_dict_overrides), so a recursive merge would hand vLLM a dict the trainer never sees.
  • Rebase note: open PR #4228 merges these two dicts inline at its grpo vLLM site (grpo.py L1543-L1548), letting the vLLM value win. Its recipe's keys don't overlap the policy's, so whichever PR lands second can take this helper without changing that recipe.

Generated by Claude Code

Comment on lines +67 to +73
if conflicts:
warnings.warn(
"policy.hf_config_overrides overrides policy.generation.vllm_kwargs."
f"hf_overrides for {conflicts}; the training policy's value is used so "
"the generation engine stays in sync with the trainer.",
stacklevel=2,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C1 · 1 action item.
TL;DR — when both override dicts set one key to different values, vLLM gets the policy value and the run goes on with only a warning in the driver log. Elsewhere the repo raises a setup-time ValueError when two user settings disagree.

For example, policy.hf_config_overrides.rope_theta: 1e6 plus policy.generation.vllm_kwargs.hf_overrides.rope_theta: 5e5: the run starts and vLLM uses 1e6. The user asked for two different values and is never asked to pick one.

Other places in the repo reject the config instead of picking a winner:

No recipe on any public branch sets one key to different values in these two dicts, so a raise blocks no existing config.

AI-1

Could you replace the warning with a ValueError that names the keys and the fix?

    if conflicts:
        raise ValueError(
            f"policy.generation.vllm_kwargs.hf_overrides sets {conflicts} to values "
            "that differ from policy.hf_config_overrides. The trainer and vLLM must "
            "load the same HF config; set these keys only in policy.hf_config_overrides."
        )

This is a plain block, not a one-click suggestion, because three other spots describe the warning and need to change with it:

Where the new error fires in distillation and single-controller setup (no action needed)

With a raise, where the call sits starts to matter:

Moving these two calls earlier, before any model is built (distillation already has an early vLLM block), would make the error appear first. Not part of this ask.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ff378e1: conflicting keys now raise your ValueError (names the keys and says to set them only in policy.hf_config_overrides). Updated the docstring, replaced the warn test with test_conflicting_values_raise (pytest.raises(ValueError, match=r"\['rope_theta'\]")), and changed the YaRN guide sentence. Equal values and generation-only keys behave as before. I also checked that no config under examples/configs or research/ sets vllm_kwargs.hf_overrides, so nothing existing trips it. Left the call-site ordering in distillation / non-colocated SC as is, per your note. Thanks for the careful review and for deferring #4949.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Oct 27, 2026
Review on NVIDIA-NeMo#4907 (C1): when policy.hf_config_overrides and
policy.generation.vllm_kwargs.hf_overrides set one key to different
values, the run started with only a driver-log warning and vLLM silently
used the policy value. Elsewhere the repo rejects disagreeing user
settings at setup (nvfp4_pertoken engine kwargs, Dynamo argv builder).

merge_policy_hf_overrides now raises a ValueError naming the keys and the
fix. Equal values and generation-only keys are unchanged. The docstring,
the conflict test and the YaRN guide are updated to match. No shipped
config sets vllm_kwargs.hf_overrides, so no existing recipe is affected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
@lonexreb

Copy link
Copy Markdown
Contributor Author

/ok to test ff378e1

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-customer Waiting on the original author to respond labels Oct 28, 2026
lonexreb and others added 2 commits November 11, 2026 22:36
NVIDIA-NeMo#4905 removed the separate dynamo generation backend, so the dynamo
branches this PR patched and their test are gone; dynamo now runs on
the vLLM path, which already merges hf_config_overrides.

Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lonexreb

Copy link
Copy Markdown
Contributor Author

@yuki-97 @RayenTian merged main. #4905 removed the separate dynamo backend, so this PR's dynamo sites are gone (description updated); the vLLM merge still covers dynamo. Needs a CI:* label and /ok to test 6cfd561752e9e3b355ee49080d2782925615e652. Thanks!

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Documentation Improvements or additions to documentation waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hf_config_overrides propagation silently discards policy.generation.vllm_kwargs.hf_overrides

3 participants