Fix warning message for PEFT models in text-generation pipeline #36783

[falconlee236]

What does this PR do?

This PR fixes the warning message when using a PeftModel with a text-generation pipeline. Currently, when a user passes a PeftModel to the text-generation pipeline, they receive an error message stating that "The model 'PeftModel' is not supported for text-generation" even though the pipeline works correctly. This PR modifies the model type checking logic in the pipeline to properly handle PEFT-wrapped models, making the user experience smoother by eliminating unnecessary warning messages.

Main Cause

in src/transformers/pipelines/text_generation.py, there Checks if the model is suitable for generation tasks.

def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.check_model_type(
            TF_MODEL_FOR_CAUSAL_LM_MAPPING_NAMES if self.framework == "tf" else MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
        )

but PEFT model didn't include neiter TF_MODEL_FOR_CAUSAL_LM_MAPPING_NAMES nor MODEL_FOR_CAUSAL_LM_MAPPING_NAMES

in check_model_type function, the peft model didn't include valid model. so logger.error was triggered. but text generation worked properly.

def check_model_type(self, supported_models: Union[List[str], dict]):
        """
        Check if the model class is in supported by the pipeline.

        Args:
            supported_models (`List[str]` or `dict`):
                The list of models supported by the pipeline, or a dictionary with model class values.
        """
        if not isinstance(supported_models, list):  # Create from a model mapping
            supported_models_names = []
            for _, model_name in supported_models.items():
                # Mapping can now contain tuples of models for the same configuration.
                if isinstance(model_name, tuple):
                    supported_models_names.extend(list(model_name))
                else:
                    supported_models_names.append(model_name)
            if hasattr(supported_models, "_model_mapping"):
                for _, model in supported_models._model_mapping._extra_content.items():
                    if isinstance(model_name, tuple):
                        supported_models_names.extend([m.__name__ for m in model])
                    else:
                        supported_models_names.append(model.__name__)
            supported_models = supported_models_names
        if self.model.__class__.__name__ not in supported_models:
            logger.error(
                f"The model '{self.model.__class__.__name__}' is not supported for {self.task}. Supported models are"
                f" {supported_models}."
            )

bug logs

python hug_inference.py 
/root/workspace/lora_test/.venv/lib/python3.10/site-packages/transformers/quantizers/auto.py:206: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.
  warnings.warn(warning_msg)
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:07<00:00,  1.12it/s]
Device set to use cuda:0
/root/workspace/lora_test/.venv/lib/python3.10/site-packages/bitsandbytes/nn/modules.py:451: UserWarning: Input type into Linear4bit is torch.float16, but bnb_4bit_compute_dtype=torch.float32 (default). This will lead to slow inference or training speed.
  warnings.warn(
this is base model result: [{'generated_text': "Who is a Elon Musk? Well, he's a business magnate, investor, and entrepreneur. He's known for his ambitious"}]
Device set to use cuda:0
The model 'PeftModel' is not supported for text-generation. Supported models are ['AriaTextForCausalLM', 'BambaForCausalLM', 'BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'BigBirdPegasusForCausalLM', 'BioGptForCausalLM', 'BlenderbotForCausalLM', 'BlenderbotSmallForCausalLM', 'BloomForCausalLM', 'CamembertForCausalLM', 'LlamaForCausalLM', 'CodeGenForCausalLM', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'CpmAntForCausalLM', 'CTRLLMHeadModel', 'Data2VecTextForCausalLM', 'DbrxForCausalLM', 'DiffLlamaForCausalLM', 'ElectraForCausalLM', 'Emu3ForCausalLM', 'ErnieForCausalLM', 'FalconForCausalLM', 'FalconMambaForCausalLM', 'FuyuForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'GitForCausalLM', 'GlmForCausalLM', 'GotOcr2ForConditionalGeneration', 'GPT2LMHeadModel', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTNeoForCausalLM', 'GPTNeoXForCausalLM', 'GPTNeoXJapaneseForCausalLM', 'GPTJForCausalLM', 'GraniteForCausalLM', 'GraniteMoeForCausalLM', 'GraniteMoeSharedForCausalLM', 'HeliumForCausalLM', 'JambaForCausalLM', 'JetMoeForCausalLM', 'LlamaForCausalLM', 'MambaForCausalLM', 'Mamba2ForCausalLM', 'MarianForCausalLM', 'MBartForCausalLM', 'MegaForCausalLM', 'MegatronBertForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'MllamaForCausalLM', 'MoshiForCausalLM', 'MptForCausalLM', 'MusicgenForCausalLM', 'MusicgenMelodyForCausalLM', 'MvpForCausalLM', 'NemotronForCausalLM', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'OlmoeForCausalLM', 'OpenLlamaForCausalLM', 'OpenAIGPTLMHeadModel', 'OPTForCausalLM', 'PegasusForCausalLM', 'PersimmonForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'PhimoeForCausalLM', 'PLBartForCausalLM', 'ProphetNetForCausalLM', 'QDQBertLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RecurrentGemmaForCausalLM', 'ReformerModelWithLMHead', 'RemBertForCausalLM', 'RobertaForCausalLM', 'RobertaPreLayerNormForCausalLM', 'RoCBertForCausalLM', 'RoFormerForCausalLM', 'RwkvForCausalLM', 'Speech2Text2ForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'TransfoXLLMHeadModel', 'TrOCRForCausalLM', 'WhisperForCausalLM', 'XGLMForCausalLM', 'XLMWithLMHeadModel', 'XLMProphetNetForCausalLM', 'XLMRobertaForCausalLM', 'XLMRobertaXLForCausalLM', 'XLNetLMHeadModel', 'XmodForCausalLM', 'ZambaForCausalLM', 'Zamba2ForCausalLM'].
this is lora model result: [{'generated_text': "Who is a Elon Musk? I mean, I know he's a business magnate or something, but what has he actually done"}]

Impelementation

So I include peft model classes in supported_models_names.

• This makes the PEFT model pass the pipeline check.

Thanks for telling me about all the types of peft models. @BenjaminBossan

I'm not very familiar with pipelines but this is what I gather: I think we should check if peft is installed, and if it is, it should be added to the supported_models list. However, I'm not sure why self.model.class.name not in supported_models is checked instead of isinstance, maybe to avoid imports? The issue with that is that we have many PeftModel subclasses, so the list would need to be extended by:
"PeftModel", "PeftModelForSequenceClassification", "PeftModelForCausalLM", "PeftModelForSeq2SeqLM", "PeftModelForTokenClassification", "PeftModelForQuestionAnswering", "PeftModelForFeatureExtraction"

resolve message

Test Description

This test verifies that the warning message "The model 'PeftModel' is not supported for text-generation" does not appear when using a PeftModel with the text-generation pipeline. The test:

1. Loads the OPTForCausalLM base model from the Hugging Face repository.
2. Loads a LoRA adapter using the PeftModel framework.
3. Sets up a text-generation pipeline with the LoRA-adapted model.
4. Runs the pipeline with a sample input text: "Who is Elon Musk?"
5. Captures log output while executing the pipeline.
6. Asserts that the expected warning message is not present in the logs.

This test ensures that PeftModel is properly integrated with the transformers pipeline without generating unnecessary warning messages.

Fixes #36783

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

cc: @sayakpaul @BenjaminBossan @Rocketknight1

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

[Rocketknight1]

Hi, I think adding PEFT support here is a good idea, but I think it would make more sense to add the model type based on self.task. For example, a PEFTForQuestionAnswering model would probably fail if used in a text-generation pipeline, so we don't want to automatically accept that.

[falconlee236]

Hi @Rocketknight1
Thank you for your review
From what I understand, you mean it looks better to check the model type based on self.task, not self.model_name?

[Rocketknight1]

Checking self.model_name is fine, but the problem is that this line:

supported_models_names = [
    "PeftModelForCausalLM",
    "PeftModelForSeq2SeqLM",
    "PeftModelForSequenceClassification",
    "PeftModelForQuestionAnswering",
    "PeftModelForTokenClassification",
    "PeftModel",
    "PeftModelForFeatureExtraction",
]

means that all PEFT model types will be accepted for all pipelines. Instead, you probably want to match model types to their appropriate pipelines. This means that you will need to set supported_model_names based on the value of self.task.

[falconlee236]

HI @Rocketknight1
in ci/circleci: test_processors, i met a error. but this error occurs by access grant of mistral model.
i cannot handle this error in myself, because this error needs to hf accesstoken in ci containers.

Cannot access gated repo for url https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/resolve/main/preprocessor_config.json

Btw,
Please check my code above? I declare CONSTANT about the kinds of support peft model.
and set model_name array based on self.task

[falconlee236]

Hi @Rocketknight1
It's been more than a week since the PR opened, can you review my PR?

Appreciate all the effort you put into reviewing others' code. It’s always a lot of work, and it doesn’t go unnoticed!

[Rocketknight1]

Minor typo but this makes sense to me now, and sorry for the delay! cc @BenjaminBossan for PEFT, are you okay with it?

[BenjaminBossan]

The fix looks good, thanks!

@falconlee236 I have added some comments on the test, especially since we should properly check the warning and not rely on print statements. Also, let's use different checkpoints. Moreover, I would suggest to move the tests to test_peft_integration.py, I don't think we need a new file for just this test.

@Rocketknight1 Please also check if my feedback makes sense, as I'm not so familiar with the testing standards in transformers.

[falconlee236]

HI @BenjaminBossan @Rocketknight1

I moved test to test_peft_integration.py and remove all unnecessary print statement in test code.

To check expected_error_msg, I use with CaptureLogger(pipeline_logger) as cl: statement.

so The test is passed when output do not contain The model 'PeftModel' is not supported text.

[BenjaminBossan]

Thanks for improving the test. I did, however, have an issue: The test would pass for me, even without your fix. I made some suggestions that should allow the test to fail without the fix.

[falconlee236]

Hi @BenjaminBossan @Rocketknight1 !
I committed again to reflect your request to modify the test file.
Thanks to you, I think it was an opportunity to review how to name the test code and Python's test function again.

[BenjaminBossan]

Thanks for working on the test, it now fails without the fix. LGTM.

[falconlee236]

Dear @Rocketknight1 ,
just following up on my pull request from last week. Please let me know if you need any additional information or changes. Looking forward to your review.
Best regards.

[Rocketknight1]

Yep, this looks good now! Will merge when tests finish, and thank you for the PR!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[falconlee236]

Yep, this looks good now! Will merge when tests finish, and thank you for the PR!

Thank you so a lot.
Happy to merged it 😊