Repository navigation
fix: merge hf_config_overrides into vllm hf_overrides rather than overwriting - #3729
Open
neodymium6 wants to merge 1 commit into
Open
neodymium6 wants to merge 1 commit into
neodymium6 wants to merge 1 commit into
Conversation
neodymium6
force-pushed
the
fix/hf-overrides-merge-in-algorithms
branch
2 times, most recently
from
August 20, 2026 13:25
ae36519 to
42715a8
Compare
…rwriting Signed-off-by: neodymium6 <104201402+neodymium6@users.noreply.github.com>
neodymium6
force-pushed
the
fix/hf-overrides-merge-in-algorithms
branch
from
August 20, 2026 14:08
42715a8 to
7780e65
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Merge
policy.hf_config_overridesinto the generation engine'shf_overridesinstead of overwriting it, so a user-suppliedpolicy.generation.vllm_kwargs.hf_overridesis no longer silently discarded.Five call sites assign it wholesale:
nemo_rl/algorithms/grpo.pynemo_rl/algorithms/grpo.pynemo_rl/algorithms/ppo.pynemo_rl/algorithms/distillation.pynemo_rl/algorithms/single_controller_utils/setup.pyvllm_kwargsis a user-facing config field, so this drops anything the user set underpolicy.generation.vllm_kwargs.hf_overrides— and because the default is{}, it dropsit even when the policy declares no overrides at all. No warning is emitted.
This is the same overwrite-vs-merge bug
_merge_fp8_kwargsfixed one layer down in theworker (#1413, reverted in #2188, re-fixed in #2904); the algorithm entry points were
never updated. This PR adds
merge_hf_overrides()next to the other vLLM config helpersand routes all five call sites through it.
The trainer/generation invariant is unchanged. The merge is shallow and the training
policy wins outright on a conflicting top-level key — a
rope_scalingdeclared by thepolicy replaces a generation-side one rather than being blended into it — so the two can
still not disagree on anything the policy declares, and the yarn
rope_scalingvalidationin
nemo_rl/models/megatron/setup.pykeeps applying unchanged. A generation-side overrideonly takes effect for keys the policy leaves unset.
Nothing resolves silently. The merged overrides are logged, so the value the engine
will actually see is visible at launch rather than only in downstream metrics. A
generation-side value the policy shadows with a different value warns; a key both sides
set identically overrides nothing and stays quiet. vLLM's callable form of
hf_overridescannot be merged with a mapping, so it is dropped with a warning rather than raising —
matching what
vllm_worker.pyalready does with a non-dicthf_overrides.The
ppo.pyanddistillation.pycall sites indexedgeneration_config["vllm_kwargs"]directly while the others used
setdefault; they now all usesetdefault, sincevllm_kwargsisNotRequiredinVllmConfig.Issues
Closes #3728
Usage
Before your PR is "Ready for review"
Pre checks:
Additional Information
tests/unit/models/generation/test_vllm_hf_overrides_merge.py, nextto the existing
test_vllm_fp8_hf_overrides.pyregression tests for the same class ofbug. Since that bug was fixed, reverted and re-fixed, the new tests also pin the call
sites with an
astcheck (in the style oftest_vllm_patches.py): every entry pointmust call
merge_hf_overrides(), and nothing innemo_rl/outside an allowlist of twofiles may replace
hf_overridesby assignment orupdate(). The scan is deliberatelypackage-wide rather than a list of today's call sites: the dynamo and single-controller
paths were added after the original three and inherited the bug, so a fixed list would
not catch the next one. The check flags all five pre-fix call sites.
hf_overridestest modules plus the unit teststhat import the changed modules —
test_grpo_checkpoint_engine.py,test_utils.py,test_ppo.py,test_distillation.py,test_vllm_quant_backend.py,test_vllm_worker_helpers.py,test_single_controller_setup.py(142 passed, 12 skippedfor GPU/modelopt).
ruffandpyreflyare clean.policy.generation.vllm_kwargsrather than adding a feature. That contract isdocumented for other keys —
docs/guides/eagle3-speculative-decoding.mdanddocs/guides/checkpoint-engine-refit.mdboth instruct users to set things undervllm_kwargs— thoughhf_overridesitself is not mentioned anywhere indocs/.