add load textual inversion embeddings to stable diffusion #2009

piEsposito · 2023-01-16T14:18:06Z

Should close #1985

HuggingFaceDocBuilderDev · 2023-01-16T14:23:42Z

The documentation is not available anymore as the PR was closed or merged.

patil-suraj

Thanks a lot for working on this @piEsposito , this will make loading embeddings very easy!

Instead of adding methods to every pipeline, we could create TextualInversionLoaderMixin class with a method load_textual_inversion_embeddings in pipelines/loaders.py, and the pipelines which can do text inversion can subclass from that mixin.

Also, left some comments below, more specifically.

We should follow the embeddings format of the textual inversion script so that we can load all the embeddings in https://huggingface.co/sd-concepts-library
We could also support loading single vector embedding from auto1111 and then extend it to multiple embeddings.

Thanks!

src/diffusers/pipelines/alt_diffusion/pipeline_alt_diffusion.py

piEsposito · 2023-01-17T12:26:39Z

@patil-suraj I'm addressing your review on the next few days, thanks!

piEsposito · 2023-01-17T13:11:48Z

@patil-suraj Github somehow un-requested review from a bunch of HF people. Can you please add them again?

patrickvonplaten · 2023-03-28T20:18:55Z

@sayakpaul @pcuenca @williamberman could you take a final look here? Made the PR now ready for diffusers design - should work for all use cases.

williamberman · 2023-03-28T20:30:12Z

tests/pipelines/stable_diffusion/test_stable_diffusion.py

+        image = pipe(
+            "An logo of a turtle in Style-Winter with <low-poly-hd-logos-icons>", generator=generator, output_type="np"
+        ).images[0]
+        # np.save("/home/patrick/diffusers-images/text_inv/winter_logo_style.npy", image)


Suggested change

# np.save("/home/patrick/diffusers-images/text_inv/winter_logo_style.npy", image)

williamberman

if we could squash and rebase on main, that would be nice.

Also assuming tests pass

sayakpaul

Let's ship this thing!

Excellent tests, btw. Let's make them pass.

pcuenca

Love the API!

src/diffusers/loaders.py

pcuenca · 2023-03-29T08:08:39Z

src/diffusers/loaders.py

+            embedding = state_dict["string_to_param"]["*"]
+
+        if token is not None and loaded_token != token:
+            logger.warn(f"The loaded token: {loaded_token} is overwritten by the passed token {token}.")


Wouldn't we want to do the opposite override? (What comes in the state_dict is what gets added)

Interesting, I'd say what gets passed has priority! If you do:

load_textual_inversion("./textual_inversion", token="<special-token>")

I think the token should be "<special-token>" no matter what's in the dict - it's similar to how we do from_pretrained(unet=unet) overrides

src/diffusers/loaders.py

src/diffusers/utils/dummy_pt_objects.py

piEsposito · 2023-03-29T14:01:51Z

Come on folks everything but an unrelated test in MPS is passing let's get this thing merged!

GuiyeC · 2023-03-29T19:35:28Z

src/diffusers/loaders.py

+            embeddings = [e for e in embedding]  # noqa: C416
+        else:
+            tokens = [token]
+            embeddings = [embedding] if len(embedding.shape) > 1 else [embedding[0]]


I was trying the latest version and I wasn't getting anything related to the embeddings, after changing this I was able to get good results. I'm not sure how this would work with len(embedding.shape) greater than 1 but at least when the shape has only one dimension this seems to fix it.

Suggested change

embeddings = [embedding] if len(embedding.shape) > 1 else [embedding[0]]

embeddings = [embedding[0]] if len(embedding.shape) > 1 else [embedding]

Suggested change

embeddings = [embedding] if len(embedding.shape) > 1 else [embedding[0]]

embeddings = [embedding] if len(embedding.shape) <= 1 else [embedding[0]]

Hmm cannot reproduce this one - my tests are passing just fine on this branch

I followed diffusers/examples/textual_inversion to train my own embedding and got learned_embeds.bin file in the end, then use the example here.

pretrained_path = 'xxx' embedding_path = 'learned_embeds.bin' pipe = DiffusionPipeline.from_pretrained(pretrained_path, torch_dtype=torch.float16) pipe.load_textual_inversion(embedding_path)

It is not working to show any concept from my enbedding token. I have to modify the same with @GuiyeC for the file src/diffusers/loaders.py to make it correctly.

I created a PR with the fix for this where I try to explain the problem a bit more.

Co-authored-by: Pedro Cuenca <[email protected]>

patrickvonplaten · 2023-03-30T17:08:35Z

Ok let's merge it 🚀

Puuh big PR - thanks so much for kickstarting this @piEsposito and for everybody involved here. Hope that the final solution / design works for everybody

EandrewJones · 2023-03-30T17:11:29Z

Thanks for getting this over the finish line guys! WIsh I could've been of more help. Great work -- I look forward to using the feature. Best Evan Jones Website: www.ea-jones.com

…

On Thu, Mar 30, 2023 at 1:08 PM Patrick von Platen ***@***.***> wrote: Merged #2009 <#2009> into main. — Reply to this email directly, view it on GitHub <#2009 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJ2T6AKXJ6GJT7NP6LQGKNLW6W42LANCNFSM6AAAAAAT4ZUCKY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

patrickvonplaten · 2023-03-30T17:12:50Z

Gosh forgot the most important part: Docs! 😅

In case someone has some spare time for a quick PR on docs on how to use it (both A1111 and diffusers) here would be some good spots to put it:

blx0102 · 2023-04-06T09:46:13Z

@piEsposito @patrickvonplaten Cheers this is done! But yes Docs are needed to show newbies like me how to use it, the discuss here is too long to read them all lol.

sayakpaul · 2023-04-06T10:43:19Z

@blx0102 would you be up for contributing a PR? :)

patrickvonplaten · 2023-04-12T10:38:08Z

Opened a quick PR here: #3068

…e#2009) * add load textual inversion embeddings draft * fix quality * fix typo * make fix copies * move to textual inversion mixin * make it accept from sd-concept library * accept list of paths to embeddings * fix styling of stable diffusion pipeline * add dummy TextualInversionMixin * add docstring to textualinversionmixin * add load textual inversion embeddings draft * fix quality * fix typo * make fix copies * move to textual inversion mixin * make it accept from sd-concept library * accept list of paths to embeddings * fix styling of stable diffusion pipeline * add dummy TextualInversionMixin * add docstring to textualinversionmixin * add case for parsing embedding from auto1111 UI format Co-authored-by: Evan Jones <[email protected]> Co-authored-by: Ana Tamais <[email protected]> * fix style after rebase * move textual inversion mixin to loaders * move mixin inheritance to DiffusionPipeline from StableDiffusionPipeline) * update dummy class name * addressed allo comments * fix old dangling import * fix style * proposal * remove bogus * Apply suggestions from code review Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Will Berman <[email protected]> * finish * make style * up * fix code quality * fix code quality - again * fix code quality - 3 * fix alt diffusion code quality * fix model editing pipeline * Apply suggestions from code review Co-authored-by: Pedro Cuenca <[email protected]> * Finish --------- Co-authored-by: Evan Jones <[email protected]> Co-authored-by: Ana Tamais <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Will Berman <[email protected]> Co-authored-by: Pedro Cuenca <[email protected]>

piEsposito added 4 commits January 16, 2023 11:15

add load textual inversion embeddings draft

6012e93

Merge branch 'main' into main

a3a800b

fix quality

d4642c7

Merge branch 'main' of github.com:piEsposito/diffusers into main

ca6d38d

piEsposito added 4 commits January 16, 2023 11:38

fix typo

c5ffdc3

Merge branch 'main' into main

32391af

make fix copies

525428d

Merge branch 'main' of github.com:piEsposito/diffusers into main

912c7c3

piEsposito marked this pull request as ready for review January 16, 2023 15:10

patil-suraj reviewed Jan 17, 2023

View reviewed changes

patil-suraj requested review from williamberman, pcuenca and patrickvonplaten January 17, 2023 10:26

piEsposito added 8 commits January 17, 2023 09:35

Merge branch 'huggingface:main' into main

15206c3

move to textual inversion mixin

fdec2d0

Merge branch 'main' of github.com:piEsposito/diffusers into main

e01a3f8

make it accept from sd-concept library

5ec8fea

accept list of paths to embeddings

5d58240

fix styling of stable diffusion pipeline

530a208

add dummy TextualInversionMixin

8e50514

add docstring to textualinversionmixin

b730987

piEsposito requested review from patil-suraj and removed request for pcuenca, williamberman and patrickvonplaten January 17, 2023 13:10

patil-suraj requested review from williamberman and pcuenca January 17, 2023 13:18

patrickvonplaten added 3 commits March 28, 2023 22:04

finish

8a040e8

make style

835a8d0

up

08a85dc

williamberman reviewed Mar 28, 2023

View reviewed changes

williamberman approved these changes Mar 28, 2023

View reviewed changes

sayakpaul approved these changes Mar 29, 2023

View reviewed changes

pcuenca approved these changes Mar 29, 2023

View reviewed changes

piEsposito added 6 commits March 29, 2023 09:19

fix code quality

d172099

fix code quality - again

991d3d7

fix code quality - 3

28c425b

fix alt diffusion code quality

df9f579

Merge branch 'main' into main

e101d9a

fix model editing pipeline

9dd0267

GuiyeC reviewed Mar 29, 2023

View reviewed changes

patrickvonplaten and others added 2 commits March 30, 2023 16:16

Apply suggestions from code review

74b1e64

Co-authored-by: Pedro Cuenca <[email protected]>

Finish

b9f53cb

patrickvonplaten merged commit a937e1b into huggingface:main Mar 30, 2023

GuiyeC mentioned this pull request Mar 31, 2023

Fix textual inversion loading #2914

Merged

NicholasKao1029 mentioned this pull request Mar 31, 2023

Embedding siliconflow/onediff#166

Closed

	embeddings = [embedding] if len(embedding.shape) > 1 else [embedding[0]]
	embeddings = [embedding[0]] if len(embedding.shape) > 1 else [embedding]

add load textual inversion embeddings to stable diffusion #2009

add load textual inversion embeddings to stable diffusion #2009

Uh oh!

Conversation

piEsposito commented Jan 16, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patil-suraj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

piEsposito commented Jan 17, 2023

Uh oh!

piEsposito commented Jan 17, 2023

Uh oh!

patrickvonplaten commented Mar 28, 2023

Uh oh!

williamberman Mar 28, 2023

Choose a reason for hiding this comment

Uh oh!

williamberman left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcuenca Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Mar 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

piEsposito commented Mar 29, 2023

Uh oh!

GuiyeC Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Mar 30, 2023

Choose a reason for hiding this comment

Uh oh!

JarvusChen Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuiyeC Mar 31, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Mar 30, 2023

Uh oh!

EandrewJones commented Mar 30, 2023 via email

Uh oh!

patrickvonplaten commented Mar 30, 2023

Uh oh!

blx0102 commented Apr 6, 2023

Uh oh!

sayakpaul commented Apr 6, 2023

Uh oh!

patrickvonplaten commented Apr 12, 2023

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jan 16, 2023 •

edited

Loading

JarvusChen Mar 31, 2023 •

edited

Loading