Skip to content

Cannot load multiple LoRAs. Need to revert PR #3649 #3781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JemiloII opened this issue Jun 14, 2023 · 7 comments
Closed

Cannot load multiple LoRAs. Need to revert PR #3649 #3781

JemiloII opened this issue Jun 14, 2023 · 7 comments
Labels
bug Something isn't working

Comments

@JemiloII
Copy link

Describe the bug

#3649 Introduced a bug. It automatically reverts any lora loaded when calling load_lora_weights. I can no longer load multiple loras. Rather than automatically unloading loras, we should have a method to unload them individually and a method to unload them all. This change has completely broke my app and I have to downgrade or revert the files in PR #3649

Reproduction

Just use load_lora_weights multiple times and note that only a single lora is in use.

Logs

No response

System Info

diffusers v0.17.1

@JemiloII JemiloII added the bug Something isn't working label Jun 14, 2023
@sayakpaul
Copy link
Member

Were you observing expected outputs with multiple LoRAs? If so, I would appreciate some concrete examples here. I am asking this because we actually don't have the technical implementation to support operating with multiple LoRAs.

Also, if you checked the PR #3649, you'd have noticed that we had to do it to avoid latching into recursive calls to monkey-patching.

So, the broader question here is supporting multiple LoRAs in a streamlined manner (which we are internally brainstorming already).

@JemiloII
Copy link
Author

JemiloII commented Jun 14, 2023

I think the reason I didn't run into the monkey patch issue is that I would load which LoRAs I wanted to use and leave them loaded in memory. I still see this as a bug as there is no option to not disable the auto-remove feature added. If I needed to change, I would restart my app. So if I'm understanding correctly, from this PR and the other I commented on.

We want the following:

  • Load LoRA & LyCORIS
  • To Unload LoRAs & LyCORIS
  • To Allow Multiple LoRAs & LyCORIS loaded
  • Does not require an app restart

Anything else I'm missing before I get started? I am already updating mine to add an option to load_lora_weights to not auto remove/monkey patch. But I can add some more features to it if it means that it can be merged into the project without me having to constantly merge updates to it.

@sayakpaul
Copy link
Member

I still see this as a bug as there is no option to not disable the auto-remove feature added

See #3689. Specifically #3689 (comment)

Regarding supporting other formats and multiple variants of them, we don't have anything immediate. We will continue to monitor it and look for opportunities here and happy to accept community contributions after some discussions with the potential contributors.

A good example is: #3437 (by @takuma104).

Cc: @patrickvonplaten

@takuma104
Copy link
Contributor

takuma104 commented Jun 14, 2023

@JemiloII If you want to use multiple LoRAs at the same time right now, feel free to use the hook version I wrote before. Although this code couldn't be directly merged into Diffusers, it's what I personally use. The usage is written in the comments, so please refer to that.
https://gist.github.com/takuma104/e38d683d72b1e448b8d9b3835f7cfa44

@JemiloII
Copy link
Author

@takuma104 Yeah, this works very well. I was able to reuse my original LoRA scales/weights from before and generate the same images.

@sayakpaul
Copy link
Member

@JemiloII thanks for trying out Takuma's version.

With #3781 (comment) in mind, maybe we can close the issue?

As mentioned, we will revisit the design to support multiple LoRAs from diffusers preferably using peft.

@JemiloII
Copy link
Author

Sounds good to me. Thanks again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants