Add bnb to the list of supported quantization methods for LLama4 #37348
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the |
src/transformers/quantizers/base.py
Outdated
| self.quantization_config.quant_method == QuantizationMethod.COMPRESSED_TENSORS | ||
| or self.quantization_config.quant_method == QuantizationMethod.BITS_AND_BYTES |
There was a problem hiding this comment.
Let's create update the mapping to add another keys about quant methods that require this change
SunMarc
left a comment
SunMarc
left a comment
There was a problem hiding this comment.
Nice, last thing is to make sure that this is restricted for now only for pre_quantized models. We can change that in the future if this is needed.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
let's do the patch attribute that i talked about before this change breaks other bnb models ! |
…gingface#37348) * add bnb * style * update * add pre_quantized check
…gingface#37348) * add bnb * style * update * add pre_quantized check
What does this PR do?
This pr adds support for bitsandbytes quantization, and loading bnb quantized models in transformers.
Some quantized Llama4 models can be found here :