Kohya LORA and variant Loading for SDXL

[CoffeeVampir3]

Currently, LoraLoaderMixin supports Koha format for older SD models. However, with the change in architecture and the two text encoders, the process is now different for SDXL.

I don't feel overly confident implementing this myself, but I have made an SDXL lora (Lycoris) that loads up fine in comfyui to test with:
https://civitai.com/models/111031?modelVersionId=119726

Cheers,
Z.

[patrickvonplaten]

cc @sayakpaul

[sayakpaul]

Thanks for bringing this to our attention.

If you want to use it diffusers right away, you could repurpose a script like the one shown here.

We're working on bringing better support for this in load_lora_weights() but it's challenging because there are so many formats to deal with. So, it will take a bit of time.

[sayakpaul]

Hmm, I was investigating the LoRA you mentioned and it's obvious that the LoRA has three model components affected: two text encoders and the UNet.

On top of that, it seems to be adding LoRA on a multitude of places:

• Convs
• FFNs in the transformer blocks
• Skip connections
  etc.

I am also seeing a new type of identifier "hada" -- what does it stand for?

From Diffusers, we only have LoRA support for the attention blocks. This is why we started #3756. As you can imagine to have a "somewhat" robust design, we need to work a bit.

[CoffeeVampir3]

Hmm, I was investigating the LoRA you mentioned and it's obvious that the LoRA ha three model components affected: two text encoders and the UNet.

On top of that, it seems to be adding LoRA on a multitude of places:

* Convs

* FFNs in the transformer blocks

* Skip connections
  etc.

I am also seeing a new type of identifier "hada" -- what does it stand for?

From Diffusers, we only have LoRA support for the attention blocks. This is why we started #3756. As you can imagine to have a "somewhat" robust design, we need to work a bit.

Thanks for your efforts really appreciate you and your teams work on this!

I investigated where the hada layers came from, it seems to be specifically a result of kohya's lycoris module:

I am unsure what is special about this, but here's essentially the only code of interest in regards to it:
https://github.com/bmaltais/kohya_ss/blob/master/tools/lycoris_utils.py#L382

I'm afraid I have nothing more of any technical value I can contribute in regard to the specifics, I couldn't find really any documentation or references to what hada is anywhere.

Cheers,
Z

[KohakuBlueleaf]

Hmm, I was investigating the LoRA you mentioned and it's obvious that the LoRA ha three model components affected: two text encoders and the UNet.
On top of that, it seems to be adding LoRA on a multitude of places:

* Convs

* FFNs in the transformer blocks

* Skip connections
  etc.

I am also seeing a new type of identifier "hada" -- what does it stand for?
From Diffusers, we only have LoRA support for the attention blocks. This is why we started #3756. As you can imagine to have a "somewhat" robust design, we need to work a bit.

Thanks for your efforts really appreciate you and your teams work on this!

I investigated where the hada layers came from, it seems to be specifically a result of kohya's lycoris module:

I am unsure what is special about this, but here's essentially the only code of interest in regards to it: https://github.com/bmaltais/kohya_ss/blob/master/tools/lycoris_utils.py#L382

I'm afraid I have nothing more of any technical value I can contribute in regard to the specifics, I couldn't find really any documentation or references to what hada is anywhere.

Cheers, Z

LyCORIS modules is from HERE
https://github.com/KohakuBlueleaf/LyCORIS

[isidentical]

I've started investigating how to support this natively on diffusers and it seems like there are two pre-requisite PRs before we can attempt to load kohya-ss style LoRAs at all:

• [SDXL DreamBooth LoRA] add support for text encoder fine-tuning #4097 which adds support for loading TE1 and TE2 LoRA layers (without it, even if we can detect the format properly; we can't load the changes to the text encoder).
• Load Kohya-ss style LoRAs with auxilary states #4147 which extends/augments the kohya-ss style LoRA support.

I think both of these PRs are very close to the finishing line (considering they are both approved) and I'd be happy to start the actual work once both of them are landed. In the meanwhile, I'll be using a custom branch which rebases on top of those two to see what other changes are required and update the progress here.

[sayakpaul]

Let me get back to this.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[John6666cat]

Any progress?
https://huggingface.co/docs/peft/package_reference/adapter_utils

[sayakpaul]

What is the issue?

[John6666cat]

https://huggingface.co/spaces/John6666/votepurchase-multiple-model/discussions/12#674b42879820aac918a8ad46
I've received a request and information, so I've come to check it out. There was also a major update to LoRA in Diffusers recently.
I think this is an issue for supporting LyCORIS, LoHA, LoCON, DoRA, etc. in Diffusers. Individual loading of these is supported by PEFT, but Diffusers does not support it as of today. LyCORIS is in high demand as it is the most common LoRA variant. I think LoRA Ease and other scripts can output LyCORIS.

I'm not sure about the progress, but I think it's mainly because of the difficulty in distinguishing between LoRA and its variants...
I don't know the exact context at the time...🤔

[asomoza]

We should always be careful with statements like this one: "LyCORIS is in high demand as it is the most common LoRA variant"

If we assume that civitai is the most used image generation models sharing website today, and you filter by lora and lycoris, loras are overwhelming more than Lycoris, so just by this, that statement is not correct.

Unless you are telling us that the lora variants are the minority and of those Lycoris is the most common one, the previous statement is not correct, also I never use Lycoris variants but all the other loras works right out of the box with diffusers.

To test this, I just picked up a random lycoris lora (by the website) and loaded it with diffusers, it worked without a problem too:

no lora	lora

I suggest that, better than just posting in a more than a year old closed issue, you open a new issue with the reproducible code snippet we can test and this should include a specific lora that triggers it.

[John6666cat]

Sorry for my poor English.😭 I meant to refer to the majority among the subspecies.

Anyway, we will check the operation again after tomorrow and try to raise a new issue if necessary.