[Community Pipeline] Add multilingual stable diffusion to community pipelines

[juancopi81]

This is the new PR for multilingual stable diffusion pipeline. In the last one the git commit history was borked, so I started this new one clean. @patrickvonplaten please let me know if this one is ok! Thank you very much :)

============================================================================

As part of #841 I thought it would be nice to have a pipeline with multilingual support and @patrickvonplaten said it was a good idea! :)

I had already opened a PR but since I made some mistake in git, it was a mess, so I decided to close the PR and start this new one.

Sorry for that and for the many commits, I did that way to have a nice story of the progress, but please let me know if this is wrong.

The code is very similar to the core stable_diffusion pipeline, the only difference is that here I am adding some code to detect the language of the prompt, and (if necessary) translate that to English and then continue the pipeline.

I also changed the "Positive / negative prompts" as explained in #1002.

Here is an example of how would it work:

from PIL import Image

import torch

from diffusers import DiffusionPipeline
from transformers import (
    pipeline,
    MBart50TokenizerFast,
    MBartForConditionalGeneration,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
device_dict = {"cuda": 0, "cpu": -1}

# helper function taken from: https://huggingface.co/blog/stable_diffusion
def image_grid(imgs, rows, cols):
    assert len(imgs) == rows*cols

    w, h = imgs[0].size
    grid = Image.new('RGB', size=(cols*w, rows*h))
    grid_w, grid_h = grid.size

    for i, img in enumerate(imgs):
        grid.paste(img, box=(i%cols*w, i//cols*h))
    return grid

# Add language detection pipeline
language_detection_model_ckpt = "papluca/xlm-roberta-base-language-detection"
language_detection_pipeline = pipeline("text-classification",
                                       model=language_detection_model_ckpt,
                                       device=device_dict[device])

# Add model for language translation
trans_tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-one-mmt")
trans_model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-one-mmt").to(device)

diffuser_pipeline = DiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    custom_pipeline="multilingual_stable_diffusion",
    detection_pipeline=language_detection_pipeline,
    translation_model=trans_model,
    translation_tokenizer=trans_tokenizer,
    revision="fp16",
    torch_dtype=torch.float16,
)

diffuser_pipeline.enable_attention_slicing()
diffuser_pipeline = diffuser_pipeline.to(device)

prompt = ["a photograph of an astronaut riding a horse", 
          "Una casa en la playa",
          "Ein Hund, der Orange isst",
          "Un restaurant parisien"]

output = diffuser_pipeline(prompt)

images = output.images

grid = image_grid(images, rows=2, cols=2)

This example produces the following image:

I could keep working on this PR (for instance, adding the information to the docs) if this is something you would like to have. I'd also love to create a Space with this Pipeline.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Thanks a lot for this pipeline - very cool! I think this could make quite a viral space to showcase stable-diffusion in different languages -> wdyt @osanseviero @apolinario

[osanseviero]

Yes! A Space for this would be amazing! @juancopi81 would you be interested in setting up a Space? We would be happy to assign it some GPUs so it runs fast ⚡

[juancopi81]

Hi @osanseviero Sure!! Thank you very much. I'd love to. I'll create the Space and let you know 😃

[juancopi81]

Hi @osanseviero I created a Space. Right now is using CPU but I am happy to change it if GPUs are provided :) I applied already for a grant. Thank you very much again!