fix a bug of prompt embeds in sdxl

[xiaohu2015]

What does this PR do?

Fixes the bug in #3995
when num_images_per_prompt > 1, it give a error

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[pcuenca]

Hi @xiaohu2015! Could you please explain the bug this solves, and how to to reproduce it?

Many thanks!

[xiaohu2015]

Hi @xiaohu2015! Could you please explain the bug this solves, and how to to reproduce it?

Many thanks!

Fixes the bug in #3995
when num_images_per_prompt > 1, it give a error

[patrickvonplaten]

Hey @xiaohu2015,

Thanks for the PR. I can't reproduce the problem with num_images_per_prompt when using the following code snippet:

from diffusers import DiffusionPipeline, EulerDiscreteScheduler, StableDiffusionPipeline, KDPM2DiscreteScheduler, StableDiffusionImg2ImgPipeline, HeunDiscreteScheduler, KDPM2AncestralDiscreteScheduler, DDIMScheduler
from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline, AutoencoderKL
import time
from pytorch_lightning import seed_everything
import os
from huggingface_hub import HfApi
# from compel import Compel
import torch
import sys
from pathlib import Path
import requests
from PIL import Image
from io import BytesIO

api = HfApi()
start_time = time.time()

use_refiner = True

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16, force_upcast=True)
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", vae=vae, torch_dtype=torch.float16, variant="fp16", use_safetensors=True, local_files_only=True)
pipe.to("cuda")

refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", vae=vae, torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
refiner.to("cuda")


prompt = "An astronaut riding a green horse on Mars"
steps = 20

image = pipe(prompt=prompt, num_inference_steps=steps, num_images_per_prompt=2, output_type="pil").images[0]
image = refiner(prompt=prompt, num_inference_steps=steps - 10, num_images_per_prompt=2, image=image).images[0]

[patrickvonplaten]

Could you maybe add a code snippet that shows the error?

[xiaohu2015]

from diffusers import DiffusionPipeline, EulerDiscreteScheduler, StableDiffusionPipeline, KDPM2DiscreteScheduler, StableDiffusionImg2ImgPipeline, HeunDiscreteScheduler, KDPM2AncestralDiscreteScheduler, DDIMScheduler
from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline, AutoencoderKL
import time
from pytorch_lightning import seed_everything
import os
from huggingface_hub import HfApi
# from compel import Compel
import torch
import sys
from pathlib import Path
import requests
from PIL import Image
from io import BytesIO

api = HfApi()
start_time = time.time()

use_refiner = True

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16, force_upcast=True)
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", vae=vae, torch_dtype=torch.float16, variant="fp16", use_safetensors=True, local_files_only=True)
pipe.to("cuda")

refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", vae=vae, torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
refiner.to("cuda")


prompt = "An astronaut riding a green horse on Mars"
negative_prompt = ""
steps = 20

prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds = pipe.encode_prompt(prompt, "cuda", 1, True, negative_prompt)

image = pipe(prompt_embeds=prompt_embeds,
        negative_prompt_embeds=negative_prompt_embeds,
        pooled_prompt_embeds=pooled_prompt_embeds,
        negative_pooled_prompt_embeds=negative_pooled_prompt_embeds, num_inference_steps=steps, num_images_per_prompt=2, output_type="pil").images[0]
#image = refiner(prompt=prompt, num_inference_steps=steps - 10, num_images_per_prompt=2, image=image).images[0]

@patrickvonplaten

[patrickvonplaten]

Great, thanks for the reproducible codesnippet! I can indeed confirm that this causes a bug! Your PR looks very nice - could you maybe just add one test here: https://github.com/huggingface/diffusers/blob/main/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl.py ?

[patrickvonplaten]

Also can you run make style and make fix-copies so that your changes are also reflected in the pipeline_stable_diffusion_inpaint.py file?

[patrickvonplaten]

Hey @xiaohu2015,

It seems like the changes break the following use case at the moment:

FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_inpaint.py::StableDiffusionXLInpaintPipelineFastTests::test_num_images_per_prompt - RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 128 but got size 64 for tensor number 1 in the list.

Could we try to make sure that the changes both work with the previous test and fix the new use case?

[xiaohu2015]

Hey @xiaohu2015,

It seems like the changes break the following use case at the moment:

FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_inpaint.py::StableDiffusionXLInpaintPipelineFastTests::test_num_images_per_prompt - RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 128 but got size 64 for tensor number 1 in the list.

Could we try to make sure that the changes both work with the previous test and fix the new use case?

thanks, it seems all work well

[patrickvonplaten]

Great! Can we now in a final step add some new tests that show that this PR fixes a bug? :-)