Skip to content

fix(nemotron-omni): keep native RADIO names in convert_single_tensor_to_hf - #3924

Merged
HuiyingLi merged 1 commit into
NVIDIA-NeMo:mainfrom
yuekaizhang:yuekai/fix-omni-radio-single-tensor-refit
Sep 17, 2026
Merged

HuiyingLi merged 1 commit into
NVIDIA-NeMo:mainfrom
yuekaizhang:yuekai/fix-omni-radio-single-tensor-refit

Conversation

@yuekaizhang

Copy link
Copy Markdown
Contributor

Problem

NemotronOmniStateDictAdapter.convert_single_tensor_to_hf (the per-tensor path used by
streaming consumers such as NeMo-RL's weight sync into vLLM) renames transformers-native
RADIO keys to the legacy checkpoint tree when vision_uses_native_radio is set:

vision_model.encoder.layer.N.attention.attention.query.weight
  -> vision_model.radio_model.model.blocks.N.attention.attention.query.weight

That name exists in neither layout. The legacy layout only has the fused
attn.qkv.{weight,bias}, and producing it needs all three of query/key/value at once,
which a single-tensor conversion never has. The whole-state-dict path (to_hf) does the
fusion correctly; the per-tensor path silently emits invalid keys for every RADIO
attention projection while the 1:1 keys (norms, MLP, patch embedding, attn.proj)
come out with valid legacy names.

Consequences for a streaming consumer:

  • a loader that only knows the legacy layout (upstream vLLM radio.py) drops all RADIO
    q/k/v weights and keeps whatever it had before, without any error;
  • a loader that also knows the native layout (the super_vl_rl vLLM fork) cannot use
    the half-renamed keys either.

Inside Automodel nothing calls this path for vision_model.*: checkpoint save and HF
export go through to_hf. The rename therefore never had a caller for which it produced
a loadable result. (supports_low_memory_dcp_load already returns False for native
RADIO for the mirror-image reason on the load side; that gate is unchanged.)

Nemotron Nano Omni is not affected: its checkpoint's remote code is the legacy timm-style
radio_model class, so the in-memory layout already matches the checkpoint and the
adapter passes vision keys through. Only Nemotron 3.5 Super VL, whose remote code is the
transformers-native RadioModel while its weights are stored in the legacy layout, hits
this path.

Change

Return the native fqn unchanged for vision_model.* in convert_single_tensor_to_hf.
A per-tensor stream then carries the same, internally consistent native names that the
in-memory module uses; consumers that understand the native layout (e.g. a vLLM RADIO
loader with a native branch, which maps attention.attention.{query,key,value} onto its
fused qkv as q/k/v shards) load all 388 RADIO parameters. Verified offline against the
Nemotron 3.5 Super VL checkpoint that the native per-tensor names cover exactly the same
vLLM parameter set as a legacy on-disk load.

to_hf is unchanged and still emits the legacy fused layout for checkpoint compatibility;
tests pin both behaviors.

Consumers that need legacy names from a stream should convert the (small) vision subtree
as a group via to_hf rather than per tensor; that is the only way to obtain the fused
attn.qkv, and it is what NeMo-RL will do for loaders without native-layout support.

Tests

  • test_convert_single_tensor_keeps_native_radio_keys – per-tensor conversion keeps native RADIO names.
  • test_to_hf_still_emits_legacy_radio_keys_for_native_radio – whole-dict export still produces the fused legacy layout.
uv run --extra automodel --with pytest pytest tests/unit_tests/models/nemotron_omni/test_nemotron_omni_state_dict_adapter.py

(21 passed on the pre-rebase base 6ae5ca23; rebased onto 23610d979 with a test-file
conflict resolved by keeping upstream's renamed supports_low_memory_dcp_load test.)

…to_hf

For transformers-native RadioModel checkpoints the legacy HF layout fuses
query/key/value into one attn.qkv tensor. The whole-state-dict path (to_hf)
performs that fusion, but convert_single_tensor_to_hf only ever sees one
tensor, so it renamed vision_model.encoder.layer.N.attention.attention.query
to the legacy radio_model.model.blocks.N.attention.attention.query -- a key
that exists in neither layout. Streaming consumers (NeMo-RL's per-tensor
refit into vLLM) silently dropped every RADIO attention weight and the vision
tower kept its previous (dummy) initialization.

Return the native fqn unchanged instead. vLLM's Super VL RADIO loader maps
native encoder.layer.N.attention.attention.{query,key,value} onto attn.qkv as
q/k/v shards, so the streamed names now cover exactly the same parameters the
legacy disk load covers (verified offline against the Nemotron 3.5 Super VL
checkpoint: 388/388 vLLM RADIO params). to_hf is unchanged and still emits the
legacy fused layout for checkpoint compatibility; tests pin both behaviors.

Signed-off-by: Yuekai <zhangyuekai@foxmail.com>
@yuekaizhang
yuekaizhang requested a review from a team as a code owner September 17, 2026 04:56
@copy-pr-bot

copy-pr-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@HuiyingLi

Copy link
Copy Markdown
Contributor

/ok to test 98ad8da

@HuiyingLi
HuiyingLi enabled auto-merge (squash) September 17, 2026 05:10
@HuiyingLi
HuiyingLi merged commit 9d875a6 into NVIDIA-NeMo:main Sep 17, 2026
81 checks passed
yuekaizhang added a commit to yuekaizhang/RL that referenced this pull request Sep 17, 2026
…refit fix)

Move the Automodel submodule from 6ae5ca23 to 9d875a6d3, which includes
NVIDIA-NeMo/Automodel#3924: NemotronOmniStateDictAdapter.convert_single_tensor_to_hf
keeps the transformers-native RADIO names instead of renaming them to a
legacy tree it cannot fuse (q/k/v -> attn.qkv), so the per-tensor refit into
the Super VL vLLM fork now covers the whole vision tower.

Regenerate uv.lock: Automodel made wandb/mlflow optional extras and
normalized its distribution name; resolved package versions are unchanged
(transformers 5.15.1, quack-kernels 0.6.4, megatron-fsdp rev unchanged).
Update the pin references in the model guide, both Super VL recipes and the
pyproject comments, and reword the refit Known Issue for the merged fix.

Signed-off-by: Yuekai <zhangyuekai@foxmail.com>

This branch was previously deployed

3 inactive deployments
public — 98ad8da9 Deployed Sep 17, 2026 by copy-pr-bot[bot] via release / finalize / notify #4632
test — 98ad8da9 Deployed Sep 17, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #10799
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants