Skip to content

[1929]: Add CLIP guidance for Img2Img stable diffusion pipeline #2723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 21, 2023

Conversation

nipunjindal
Copy link
Contributor

This PR adds the capability to enable CLIP guidance for Img2Img.

Issue: #1929
Inspired by: #561

prompt = "fantasy book cover, full moon, fantasy forest landscape, golden vector elements, fantasy magic, dark light night, intricate, elegant, sharp focus, illustration, highly detailed, digital painting, concept art, matte, art by WLOP and Artgerm and Albert Bierstadt, masterpiece"

Init Image

img2img_init_clip_guidance

Output Image

img2img_clip_guidance

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Mar 17, 2023

The documentation is not available anymore as the PR was closed or merged.

@nipunjindal
Copy link
Contributor Author

Unrelated test failing
=========================== short test summary info ============================
FAILED tests/test_unet_2d_blocks.py::SimpleCrossAttnUpBlock2DTests::test_output - AssertionError: Max diff is absolute 1.2463128566741943. Diff tensor is tensor([0.1684, 0.4059, 0.3010, 0.2362, 1.2463, 1.0658, 0.9764, 0.6841, 0.3259],
device='mps:0').
FAILED tests/test_unet_2d_blocks.py::AttnUpDecoderBlock2DTests::test_output - AssertionError: Max diff is absolute 1.7274205684661865. Diff tensor is tensor([1.0407, 0.7878, 0.0026, 1.7274, 0.4588, 0.6572, 0.2625, 0.2807, 0.1914],
device='mps:0').
===== 2 failed, 953 passed, 619 skipped, 265 warnings in 578.05s (0:09:38) =====

@nipunjindal
Copy link
Contributor Author

Here is the PR(#2717) for inconsistent test.

@patrickvonplaten
Copy link
Contributor

Looks great - thanks!

@patrickvonplaten patrickvonplaten merged commit 0042efd into huggingface:main Mar 21, 2023
@dblunk88
Copy link
Contributor

Awesome!!!

w4ffl35 pushed a commit to w4ffl35/diffusers that referenced this pull request Apr 14, 2023
…ingface#2723)

* [Img2Img]: Copyover img2img pipeline

* [Img2Img]: img2img pipeline

* [Img2Img]: img2img pipeline

* [Img2Img]: img2img pipeline

---------

Co-authored-by: njindal <[email protected]>
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
…ingface#2723)

* [Img2Img]: Copyover img2img pipeline

* [Img2Img]: img2img pipeline

* [Img2Img]: img2img pipeline

* [Img2Img]: img2img pipeline

---------

Co-authored-by: njindal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants