Please create simple way to load new embeddings on Stable Diffusion Pipeline

[piEsposito]

Is your feature request related to a problem? Please describe.
It is frustrating to load Textual Inversion embeddings on SD pipelines, adds bloat to the code. It would be good if we could just pipeline.load_embeddings and add new embeddings ad-hoc to the pipeline.

Describe the solution you'd like
I would like to implement a method on Stable Diffusion pipelines to let people load_embeddings and append them to ones from the text encoder and tokenizer, something like:

• pipeline.load_embeddings({"emb1": "emb1.ckpt"}), then
• Embedding is loaded and appended to the embedding matrix of text encoder. Token is added to tokenizer.
• Predictions work with the new token smoothly.

We could even hack a way to load tokens with multiple embedding columns, alike to what we can do on Auto1111.

Describe alternatives you've considered

• Do that by hand every single time, but then if would bloat my code and wouldn't benefit the Diffusers community
• Don't use textual inversion at all, but this would be no-fun.

Additional context
I can implement it and open a PR if you approve the idea.

[patrickvonplaten]

Hey @piEsposito,

Yes I think this makes sense! Also related is LoRA: #1884, since there we also need to load just part of the weights.

Intuitively I think the cleanest API here would be to have a "TextualInversionLoaderMixin" class that implements a

load_textual_inversion_embes(...)

function. This Mixin could then be shared with multiple pipelines. Similarly we could create a "LoRALoaderMixin".

More specifically regarding the loading, I think we should make use of the following functions:

1. pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer) + 1)
2. We should add a general set_weights(...) functions that takes a dict "{component: key: tensor}" that can set arbitrary weights.

What do you think @piEsposito ?

Also cc @apolinario @keturn @patil-suraj @anton-l @williamberman @pcuenca in case you have some nice ideas / feedback.

[piEsposito]

I’m on it. Em sex., 13 de jan. de 2023 às 11:27, Patrick von Platen < ***@***.***> escreveu:

Hey @piEsposito <https://github.com/piEsposito>, Yes I think this makes sense! Also related is LoRA: #1884 <#1884>, since there we also need to load just part of the weights. Intuitively I think the cleanest API here would be to have a " TextualInversionLoaderMixin" class that implements a load_textual_inversion_embes(...) function. This Mixin could then be shared with multiple pipelines. Similarly we could create a "LoRALoaderMixin". More specifically regarding the loading, I think we should make use of the following functions: 1. pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer) + 1) 2. We should add a general set_weights(...) functions that takes a dict "{component: key: tensor}" that can set arbitrary weights. What do you think @piEsposito <https://github.com/piEsposito> ? Also cc @apolinario <https://github.com/apolinario> @keturn <https://github.com/keturn> @patil-suraj <https://github.com/patil-suraj> @anton-l <https://github.com/anton-l> @williamberman <https://github.com/williamberman> @pcuenca <https://github.com/pcuenca> in case you have some nice ideas / feedback. — Reply to this email directly, view it on GitHub <#1985 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLYRXQOOK4OZQMZGYAGPHDWSFQ65ANCNFSM6AAAAAATZX5BV4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- -- *Pi Esposito | piesposi.to <https://piesposi.to>*

[patrickvonplaten]

@piEsposito - note I've just added a very similar functionality for LoRA: #1884 (comment) -> would be great if we could align the two loading classes API-wise.

[keturn]

InvokeAI's current code for this is somewhere around here: https://github.com/invoke-ai/InvokeAI/blob/1e2f8710b7ee508d3120450acb38a8a68e87801f/ldm/modules/textual_inversion_manager.py#L74

related discussions also on #1597 and #799 (comment)

[piEsposito]

@keturn thank you for the code sample, it helped at lot.

@patrickvonplaten what do you think of the PR? It is working and it is very simple.

[piEsposito]

Should I make it compatible with auto111 embeddings as well?

[patil-suraj]

Hey @piEsposito thanks a lot! Taking a look at the PR now.

Should I make it compatible with auto111 embeddings as well?

Yes, we could allow the loading of single vector embeddings from auto1111. Need to check the logic for how multiple vectors are handled to add that in diffusers.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[keturn]

uh, bot, why did you close this as completed?

[pcuenca]

This is being actively worked on in #2009, I hope we can merge it soon :)

[patrickvonplaten]

Sorry, very late on this one :-/ Hope to tackle this or next week