Add T5 GGUF loading support

[junejae]

What does this PR do?

Add T5 GGUF loading support

Due to the nature of T5's architecture, I decided to replicate gguf's conversion logic, so the final code gets messy.
I tried to avoid any logical conflicts between t5's and existing model architectures, but feel free to edit codes if you find any mistakes that I haven't noticed out.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case. Link: Community contribution: Adding GGUF support for more architectures #33260
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @LysandreJik @ArthurZucker , could you please review this PR?

[bonlime]

Does your code allow loading T5 Encoder XXL (the one used in Flux)?
Example files here:
https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/tree/main

[junejae]

@bonlime Yes, it works with AutoModelForTextEncoding class. I've tested with T5 encoder of the exact repo you linked, but I didn't commit the independent test code block for T5 encoder since the code could be dirty with embedding vector style example output.

[SunMarc]

Thanks for your work @junejae and sorry for the delay ! Just a few nits. There are a few merge conflits, can you fix them also ?

[junejae]

@SunMarc
I've resolved conflicts and added more tests. Could you please review it again?

[SunMarc]

Make sure to fix the CI also !

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

Nice, thanks a lot !

[LysandreJik]

Boom, awesome! Thanks @junejae

[mirix]

Is the commit merged? I still get the error:

ValueError: GGUF model with architecture t5encoder is not supported yet.

t5_gguf = 'city96/t5-v1_1-xxl-encoder-gguf'
t5_file = 't5-v1_1-xxl-encoder-Q8_0.gguf'

text_encoder_2 = T5EncoderModel.from_pretrained(
    t5_gguf,
    gguf_file=t5_file,
    torch_dtype=torch.bfloat16,
)

[SunMarc]

cc @Isotr0py

[SlimRG]

Pls, add this example into comments into UMT5EncoderModel's code
Using from_pretrained except from_single_file was new for me

Is the commit merged? I still get the error:

ValueError: GGUF model with architecture t5encoder is not supported yet.

t5_gguf = 'city96/t5-v1_1-xxl-encoder-gguf'
t5_file = 't5-v1_1-xxl-encoder-Q8_0.gguf'

text_encoder_2 = T5EncoderModel.from_pretrained(
    t5_gguf,
    gguf_file=t5_file,
    torch_dtype=torch.bfloat16,
)

[zwukong]

too slow😭.
need to convert first
Converting and de-quantizing GGUF tensors...: 100%|██████████████████████████████████████████████████████| 219/219 [00:56<00:00, 3.89it/s]

[SunMarc]

Yeah, we need to make this multi-threaded to go faster ;D

[zwukong]

comfyui is much faster, don't need to convert in each running.For some reason,i have to test gguf in diffusers😄