Fix convert to original state dict for VLMs

[hiyouga]

What does this PR do?

#37033 introduces the base models for all VLMs. The model weights will be converted by mapping the original keys according to

transformers/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py

Lines 1735 to 1738 in 701caef

	_checkpoint_conversion_mapping = {
	"^visual": "model.visual",
	r"^model(?!\.(language_model|visual))": "model.language_model",
	}

However, the previous implementation of Transformers could not properly convert the weights back due to existing bugs (replacement = re.sub(r"\(.*?\)", "", pattern) -> replacement = re.sub(r"\(.*?\)", "", replacement)) and the lack of support for nested parentheses

transformers/src/transformers/modeling_utils.py

Lines 3644 to 3657 in 701caef

	if any(allowed_name in self.__class__.__name__.lower() for allowed_name in VLMS):
	reverse_key_mapping = {v: k for k, v in self._checkpoint_conversion_mapping.items()}
	original_state_dict = {}
	for key, value in state_dict.items():
	for pattern, replacement in reverse_key_mapping.items():
	replacement = replacement.lstrip("^") # strip off un-needed chars and patterns
	replacement = re.sub(r"\(.*?\)", "", pattern)
	key, n_replace = re.subn(pattern, replacement, key)
	# Early exit of the loop
	if n_replace > 0:
	break
	original_state_dict[key] = value
	state_dict = original_state_dict

We want to provide a more accurate weight conversion implementation to prevent issues with third-party apps.
hiyouga/LLaMA-Factory#8147

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@ArthurZucker @zucchini-nlp

[zucchini-nlp]

thanks

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hiyouga]

@zucchini-nlp Thanks for the merging. Could we have a unittest for this function?

[zucchini-nlp]

I don't think we need a test, since it's fixed already and we probably won't touch that part anymore. We have a test for loading the ckpt and for creating a new dummy model, loading-saving-loading-saving it back. That is enough imo

[hiyouga]

@zucchini-nlp Okay. By the way, do you think we should override the state_dict and named_parameters method to ensure consistency between the output and the original model weights? For example, OpenR1 uses the named_parameters method to synchronize the model weights with the inference engine.

https://github.com/huggingface/trl/blob/fef915e36f12f759b384e4ab6f650208130aa232/trl/trainer/grpo_trainer.py#L872-L875