Repository navigation
fix(nemotron-omni): keep native RADIO names in convert_single_tensor_to_hf - #3924
Merged
HuiyingLi merged 1 commit intoSep 17, 2026
Conversation
…to_hf
For transformers-native RadioModel checkpoints the legacy HF layout fuses
query/key/value into one attn.qkv tensor. The whole-state-dict path (to_hf)
performs that fusion, but convert_single_tensor_to_hf only ever sees one
tensor, so it renamed vision_model.encoder.layer.N.attention.attention.query
to the legacy radio_model.model.blocks.N.attention.attention.query -- a key
that exists in neither layout. Streaming consumers (NeMo-RL's per-tensor
refit into vLLM) silently dropped every RADIO attention weight and the vision
tower kept its previous (dummy) initialization.
Return the native fqn unchanged instead. vLLM's Super VL RADIO loader maps
native encoder.layer.N.attention.attention.{query,key,value} onto attn.qkv as
q/k/v shards, so the streamed names now cover exactly the same parameters the
legacy disk load covers (verified offline against the Nemotron 3.5 Super VL
checkpoint: 388/388 vLLM RADIO params). to_hf is unchanged and still emits the
legacy fused layout for checkpoint compatibility; tests pin both behaviors.
Signed-off-by: Yuekai <zhangyuekai@foxmail.com>
HuiyingLi
approved these changes
Sep 17, 2026
Contributor
|
/ok to test 98ad8da |
HuiyingLi
enabled auto-merge (squash)
September 17, 2026 05:10
yuekaizhang
added a commit
to yuekaizhang/RL
that referenced
this pull request
Sep 17, 2026
…refit fix) Move the Automodel submodule from 6ae5ca23 to 9d875a6d3, which includes NVIDIA-NeMo/Automodel#3924: NemotronOmniStateDictAdapter.convert_single_tensor_to_hf keeps the transformers-native RADIO names instead of renaming them to a legacy tree it cannot fuse (q/k/v -> attn.qkv), so the per-tensor refit into the Super VL vLLM fork now covers the whole vision tower. Regenerate uv.lock: Automodel made wandb/mlflow optional extras and normalized its distribution name; resolved package versions are unchanged (transformers 5.15.1, quack-kernels 0.6.4, megatron-fsdp rev unchanged). Update the pin references in the model guide, both Super VL recipes and the pyproject comments, and reword the refit Known Issue for the merged fix. Signed-off-by: Yuekai <zhangyuekai@foxmail.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
NemotronOmniStateDictAdapter.convert_single_tensor_to_hf(the per-tensor path used bystreaming consumers such as NeMo-RL's weight sync into vLLM) renames transformers-native
RADIO keys to the legacy checkpoint tree when
vision_uses_native_radiois set:That name exists in neither layout. The legacy layout only has the fused
attn.qkv.{weight,bias}, and producing it needs all three of query/key/value at once,which a single-tensor conversion never has. The whole-state-dict path (
to_hf) does thefusion correctly; the per-tensor path silently emits invalid keys for every RADIO
attention projection while the 1:1 keys (norms, MLP, patch embedding,
attn.proj)come out with valid legacy names.
Consequences for a streaming consumer:
radio.py) drops all RADIOq/k/v weights and keeps whatever it had before, without any error;
super_vl_rlvLLM fork) cannot usethe half-renamed keys either.
Inside Automodel nothing calls this path for
vision_model.*: checkpoint save and HFexport go through
to_hf. The rename therefore never had a caller for which it produceda loadable result. (
supports_low_memory_dcp_loadalready returnsFalsefor nativeRADIO for the mirror-image reason on the load side; that gate is unchanged.)
Nemotron Nano Omni is not affected: its checkpoint's remote code is the legacy timm-style
radio_modelclass, so the in-memory layout already matches the checkpoint and theadapter passes vision keys through. Only Nemotron 3.5 Super VL, whose remote code is the
transformers-native
RadioModelwhile its weights are stored in the legacy layout, hitsthis path.
Change
Return the native fqn unchanged for
vision_model.*inconvert_single_tensor_to_hf.A per-tensor stream then carries the same, internally consistent native names that the
in-memory module uses; consumers that understand the native layout (e.g. a vLLM RADIO
loader with a native branch, which maps
attention.attention.{query,key,value}onto itsfused qkv as q/k/v shards) load all 388 RADIO parameters. Verified offline against the
Nemotron 3.5 Super VL checkpoint that the native per-tensor names cover exactly the same
vLLM parameter set as a legacy on-disk load.
to_hfis unchanged and still emits the legacy fused layout for checkpoint compatibility;tests pin both behaviors.
Consumers that need legacy names from a stream should convert the (small) vision subtree
as a group via
to_hfrather than per tensor; that is the only way to obtain the fusedattn.qkv, and it is what NeMo-RL will do for loaders without native-layout support.Tests
test_convert_single_tensor_keeps_native_radio_keys– per-tensor conversion keeps native RADIO names.test_to_hf_still_emits_legacy_radio_keys_for_native_radio– whole-dict export still produces the fused legacy layout.(21 passed on the pre-rebase base
6ae5ca23; rebased onto23610d979with a test-fileconflict resolved by keeping upstream's renamed
supports_low_memory_dcp_loadtest.)