Skip to content

fix: merge hf_config_overrides into vllm hf_overrides rather than overwriting - #3729

Open
neodymium6 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
neodymium6:fix/hf-overrides-merge-in-algorithms
Open

neodymium6 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
neodymium6:fix/hf-overrides-merge-in-algorithms

Conversation

@neodymium6

@neodymium6 neodymium6 commented Aug 20, 2026 •

Copy link
Copy Markdown

What does this PR do ?

Merge policy.hf_config_overrides into the generation engine's hf_overrides instead of overwriting it, so a user-supplied policy.generation.vllm_kwargs.hf_overrides is no longer silently discarded.

Five call sites assign it wholesale:

file line backend
nemo_rl/algorithms/grpo.py 1345 vllm
nemo_rl/algorithms/grpo.py 496 dynamo
nemo_rl/algorithms/ppo.py 817 vllm
nemo_rl/algorithms/distillation.py 514 vllm
nemo_rl/algorithms/single_controller_utils/setup.py 227 vllm
## make vllm hf overrides match the training policy
vllm_kwargs["hf_overrides"] = policy_config.get("hf_config_overrides", {})

vllm_kwargs is a user-facing config field, so this drops anything the user set under
policy.generation.vllm_kwargs.hf_overrides — and because the default is {}, it drops
it even when the policy declares no overrides at all. No warning is emitted.

This is the same overwrite-vs-merge bug _merge_fp8_kwargs fixed one layer down in the
worker (#1413, reverted in #2188, re-fixed in #2904); the algorithm entry points were
never updated. This PR adds merge_hf_overrides() next to the other vLLM config helpers
and routes all five call sites through it.

The trainer/generation invariant is unchanged. The merge is shallow and the training
policy wins outright on a conflicting top-level key — a rope_scaling declared by the
policy replaces a generation-side one rather than being blended into it — so the two can
still not disagree on anything the policy declares, and the yarn rope_scaling validation
in nemo_rl/models/megatron/setup.py keeps applying unchanged. A generation-side override
only takes effect for keys the policy leaves unset.

Nothing resolves silently. The merged overrides are logged, so the value the engine
will actually see is visible at launch rather than only in downstream metrics. A
generation-side value the policy shadows with a different value warns; a key both sides
set identically overrides nothing and stays quiet. vLLM's callable form of hf_overrides
cannot be merged with a mapping, so it is dropped with a warning rather than raising —
matching what vllm_worker.py already does with a non-dict hf_overrides.

The ppo.py and distillation.py call sites indexed generation_config["vllm_kwargs"]
directly while the others used setdefault; they now all use setdefault, since
vllm_kwargs is NotRequired in VllmConfig.

Issues

Closes #3728

Usage

policy:
  # no hf_config_overrides
  generation:
    backend: vllm
    vllm_kwargs:
      hf_overrides:
        max_position_embeddings: 8192   # before: silently reset to {}; after: reaches vLLM

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • New tests live in tests/unit/models/generation/test_vllm_hf_overrides_merge.py, next
    to the existing test_vllm_fp8_hf_overrides.py regression tests for the same class of
    bug. Since that bug was fixed, reverted and re-fixed, the new tests also pin the call
    sites with an ast check (in the style of test_vllm_patches.py): every entry point
    must call merge_hf_overrides(), and nothing in nemo_rl/ outside an allowlist of two
    files may replace hf_overrides by assignment or update(). The scan is deliberately
    package-wide rather than a list of today's call sites: the dynamo and single-controller
    paths were added after the original three and inherited the bug, so a fixed list would
    not catch the next one. The check flags all five pre-fix call sites.
  • Ran locally (CPU): the new and existing hf_overrides test modules plus the unit tests
    that import the changed modules — test_grpo_checkpoint_engine.py, test_utils.py,
    test_ppo.py, test_distillation.py, test_vllm_quant_backend.py,
    test_vllm_worker_helpers.py, test_single_controller_setup.py (142 passed, 12 skipped
    for GPU/modelopt). ruff and pyrefly are clean.
  • No documentation change: this restores the pass-through contract of
    policy.generation.vllm_kwargs rather than adding a feature. That contract is
    documented for other keys — docs/guides/eagle3-speculative-decoding.md and
    docs/guides/checkpoint-engine-refit.md both instruct users to set things under
    vllm_kwargs — though hf_overrides itself is not mentioned anywhere in docs/.

@neodymium6
neodymium6 requested review from a team as code owners August 20, 2026 12:47
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@neodymium6
neodymium6 force-pushed the fix/hf-overrides-merge-in-algorithms branch 2 times, most recently from ae36519 to 42715a8 Compare August 20, 2026 13:25
…rwriting

Signed-off-by: neodymium6 <104201402+neodymium6@users.noreply.github.com>
@neodymium6
neodymium6 force-pushed the fix/hf-overrides-merge-in-algorithms branch from 42715a8 to 7780e65 Compare August 20, 2026 14:08
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Aug 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hf_config_overrides propagation silently discards policy.generation.vllm_kwargs.hf_overrides

2 participants