[VLMs] fix flash-attention tests

[zucchini-nlp]

What does this PR do?

FA2 with layer norm upcasted was failing for all VLMs before. The issue lies in VLM config structure which is composite, and modifying base config doesn't update sub-configs (which are also deepcopied in model with XXX._from_config)

The solution is to apply recursively config update on model's children, whenever a PretrainedModel is found. Though I am not sure if we need any recursive calls when prepare_model_for_quantization, it doesn't seem to update config

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[zucchini-nlp]

run-slow: janus

[github-actions]

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/janus']
quantizations: [] ...

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

SGTM ! We pop_pre_quantization_dtype when saving the config, this is why you don't see it after saving the model. I think we should also update it recursively !

[BenjaminBossan]

Hi @zucchini-nlp this PR introduced a regression in PEFT. Before the PR, the model card would look like this:

---
library_name: peft
base_model: facebook/opt-125m
---

# Model Card for Model ID
[...]

Afterwards, the base_model meta info is missing:

---
library_name: peft
---

# Model Card for Model ID
[...]

Here is a reproducer:

# testfile.py
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model_id = "facebook/opt-125m"

def test_save_peft_modelcard(tmp_path):
    model = AutoModelForCausalLM.from_pretrained(model_id)
    config = LoraConfig()
    model = get_peft_model(model, config)
    model.save_pretrained(tmp_path, safe_serialization=False)

    with open(f"{tmp_path}/README.md") as f:
        model_card = f.read()
    assert "base_model: facebook/opt-125m" in model_card

[SunMarc]

probably due to that, we don't remove this in to_dict() cc @zucchini-nlp

[zucchini-nlp]

oops, sorry, will revert that