(SDXL) load lora error

[zbulrush]

Describe the bug

pipe.load_lora_weights(".", model_path)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 137, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'StableDiffusionXLPipeline' object has no attribute 'load_lora_weights'

Reproduction

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16)

pipe.scheduler = DPMSolverMultistepScheduler.from_config(
pipe.scheduler.config, use_karras_sigmas=True
)

model_path = "./models/last-000010.safetensors"
pipe.load_lora_weights(".", model_path)

Logs

No response

System Info

Name: diffusers
Version: 0.18.2

ubuntu, python3.10.8

Who can help?

No response

[patrickvonplaten]

Here we go with SDXL and Loras haha, @zbulrush where did you take the LoRA from / how did you train it?

[patrickvonplaten]

cc @sayakpaul

[zbulrush]

Here we go with SDXL and Loras haha, @zbulrush where did you take the LoRA from / how did you train it?

I was trained using the latest version of kohya_ss. The base mode is lsdxl, and it can work well in comfyui.

I want to do more custom development. If necessary, I can provide the LoRa file.

[sayakpaul]

If you install diffusers from source (pip install git+https://github.com/huggingface/diffusers) you should have access to load_lora_weights() method from the SDXL pipeline.

But please be aware of the following:
#3725 (comment)

If you load your LoRA file, I am almost certain that you will notice that there are keys and parameters not yet supported by Diffusers. We're working on it here: #3756.

[zbulrush]

pipe.load_lora_weights(".", "last-000010.safetensors")
TypeError: LoraLoaderMixin.load_lora_weights() takes 2 positional arguments but 3 were given

[sayakpaul]

Could you try with

pipe.load_lora_weights(".", weight_name ="last-000010.safetensors")

?

[zbulrush]

Could you try with

pipe.load_lora_weights(".", weight_name ="last-000010.safetensors")

pipe.load_lora_weights(".", weight_name ="last-000010.safetensors")
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 827, in load_lora_weights
self.load_lora_into_unet(state_dict, network_alpha=network_alpha, unet=self.unet)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 1002, in load_lora_into_unet
unet.load_attn_procs(state_dict)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 321, in load_attn_procs
rank = value_dict["to_k_lora.down.weight"].shape[0]
KeyError: 'to_k_lora.down.weight'

Then I tried another model(https://civitai.com/models/106582/aogamisdxl), and it returned the following error. Both of these models can work on ComfyUI.

pipe.load_lora_weights(".", weight_name ="aogami_v4_12800_style.safetensors")
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 827, in load_lora_weights
self.load_lora_into_unet(state_dict, network_alpha=network_alpha, unet=self.unet)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 995, in load_lora_into_unet
unet.load_attn_procs(unet_lora_state_dict, network_alpha=network_alpha)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 326, in load_attn_procs
attn_processor = getattr(attn_processor, sub_key)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/models/modeling_utils.py", line 186, in getattr
return super().getattr(name)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'UNet2DConditionModel' object has no attribute 'input'

[sayakpaul]

Then it seems to be a new format altogether. So, I would like to kindly draw your attention to my comment here: #3725 (comment).

There are numerous LoRA formats in the community, and it's really painful for us to attempt to support all of them.

[zbulrush]

enh, I think the lora based on sdxl could be a whole new topic, The structure is different from the previous.

looking forward to update

[adhikjoshi]

Then it seems to be a new format altogether. So, I would like to kindly draw your attention to my comment here: #3725 (comment).

There are numerous LoRA formats in the community, and it's really painful for us to attempt to support all of them.

Can we have some list of supported lora formats?

It's getting confusing now, there's lora, lorcois, loha, lora without text encoder and list goes on.

Or maybe a function to find out if given lora is which format? Even if it doesn't support

[isidentical]

Or maybe a function to find out if given lora is which format? Even if it doesn't support

This would be super useful for real world diffusers integrations. We are currently building a generic (any base model / checkpoint) SD text to image API with support for any loras that can be uploaded on top of existing diffusers pipelines and utilities. But unfortunately there is no easy way for us to know whether a LoRA is going to work, and if it does with what precision (e.g. unloaded keys).

For a very naive version of this, we started writing an internal script but it is mainly heruistic-based for the examples we saw out in the wild and might not be %100 accurate (e.g. it matches every Conv LoRA to LyCORIS): https://gist.github.com/isidentical/b9133961df0a72109b1bcbcece1a25b7.

Since there is no standardized metadata, it is going to be super hard to keep track of it without a centralized library like diffusers providing the detection support. I'd really love a guess_lora_type() function or something like that (or better warnings/errors when LoRAs can not be loaded due to incompatibility).

[sayakpaul]

This would be super useful for real world diffusers integrations. We are currently building a generic (any base model / checkpoint) SD text to image API with support for any loras that can be uploaded on top of existing diffusers pipelines and utilities. But unfortunately there is no easy way for us to know whether a LoRA is going to work, and if it does with what precision (e.g. unloaded keys).

Would love a contribution on this. I think it should be possible to base its logic from

diffusers/src/diffusers/loaders.py

Line 965 in 8b18cd8

if all((k.startswith("lora_te_") or k.startswith("lora_unet_")) for k in state_dict.keys()):

Since there is no standardized metadata, it is going to be super hard to keep track of it

Completely empathize with this and something we're also struggling with unfortunately :(

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.