CLIP Guided Image2Image

[dblunk88]

The CLIP guided text2img is pretty good. Would be nice to be able to use it for img2img

[patrickvonplaten]

cc @patil-suraj,

Would be great to make this a community pipeline !

[akshay-babbar]

Hi @patrickvonplaten , since it will be my first issue ever, i will be happy to take this up if someone can guide me.
Can this be assigned to me?

[patrickvonplaten]

Hey @akshay-babbar,

sure would you like to open a PR for it?

[akshay-babbar]

Yes @patrickvonplaten i would like to, but from the above , it is not clear to me what needs to be done exactly? Can you explain? Is this a reference point?

[patrickvonplaten]

Hey @akshay-babbar,

The goal here is to make a community pipeline (see: #841)
that does img2img and is guided by CLIP, so quite similar to: https://github.com/huggingface/diffusers/blob/main/examples/community/clip_guided_stable_diffusion.py

[akshay-babbar]

Sure @patrickvonplaten , will be a good learning for me, can you assign this to me?

[patil-suraj]

Hey @akshay-babbar thanks, just assigned it to you, let me know if you need any help :)

[akshay-babbar]

Thanks @patil-suraj

[ryx19th]

Hi may I ask if any update here?

[nipunjindal]

Hello @patrickvonplaten,

I'd like to take it up and raise a PR.
Before moving ahead can you clarify my understanding, I'd have to add another pipeline in the community (as suggested in #841) with additional CLIP guidance?
For the Img2Img pipeline, do I inherit from https://github.com/huggingface/diffusers/blob/v0.14.0/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L93 or should I duplicate the code and mark duplicate as I read on other threads.

Would love to get clarification to enable faster convergence.

[akshay-babbar]

Hey @patil-suraj, Sorry but I was/am keeping busy, so could not get back to this. So for now please feel free to assign this issue to someone else.
Thanks.

[nipunjindal]

Hello @patil-suraj @patrickvonplaten,

Here is the PR(#2723) for CLIP guidance for Img2Img pipeline. Let me know your thoughts.

[nipunjindal]

Hello @patil-suraj @patrickvonplaten,

Sorry for the ping, were you able to take a look at the PR and would you have some initial comments?

[patrickvonplaten]

Merged! Well done, looks great!

[dblunk88]

wohoo! Awesome work! Thank you for the awesome feature