Differential Diffusion: Giving Each Pixel Its Strength

[exx8]

Model/Pipeline/Scheduler description

Hello,
I would like to suggest merging my paper: Differential Diffusion: Giving Each Pixel Its Strength.
The paper allows a user to edit a picture by a change map that describes how much each region should change.
The editing process is typically guided by textual instructions, although it can also be applied without guidance.
We support both continuous and discrete editing.
Our framework is training and fine tuning free! And has negligible penalty of the inference time.
Our implementation is diffusers-based.
We already tested it on 4 different diffusion models (Kadinsky, DeepFloyd IF, SD, SD XL).
We are confident that the framework can also be ported to other diffusion models, such as SD Turbo, Stable Cascade, and amused.
I notice that you usually stick to white==change convention, which is opposite to the convention we used in the paper.
The paper can be thought of as a generalization to some of the existing techniques.
A black map is just regular txt2img ("0"),
A map of one color (which isn't black) can be thought as img2img,
A map of two colors which one color is white can be thought as inpaint.
And the rest? It's completely new!
In the paper, we suggest some further applications such as soft inpainting and strength visualization.

Open source status

• The model implementation is available.

Provide useful links for the implementation

Site:
https://differential-diffusion.github.io/
Paper:
https://differential-diffusion.github.io/paper.pdf
Repo:
https://github.com/exx8/differential-diffusion

[asomoza]

I was waiting for the code, thank you for your work, I will try this with a diffusers only solution and see how good are the results, they really look promising.

[yiyixuxu]

thanks for the message
we're keeping an eye on this!

[vladmandic]

if you want to test it out, i've integrated it with diffuser pipeline in sdnext and really liking the results so far.

[asomoza]

I've been testing this and is really impressive. These are some tests I did, I'll compare it to the default diffusers inpainting (with mask blur) and with fooocus since people say is the SOTA at the moment with SDXL.

Input Image	diffusers mask	differential mask

prompt: a crow on top of a branch

With diffusers, I had to lower the strength to 0.8:

Diffusers

Fooocus

Differential Diffusion

I picked the best results for all three of them, with differential I did a 1.0 strength and you can see it since the crow is really black like in fooocus. The middle result from differential would be really hard (at least for my eyes) to notice that's an inpainting, except for the missing talons.

Also the demo is really cool, with a depth map you can gradually transform a scene like this:

input	default result	with marigold

My vote is that this should be added as a core option for the inpainting pipeline, this results are really good, at least for me and would make diffusers be as good as other solutions.

[vladmandic]

one more quick note - i like two tunable params to play with here - denoising strength as usual, but also applying relative mask brightness (-1..1), it gives so much freedom on how strong is mask being applied.

[a-r-r-o-w]

I've been playing around with this as well and the results are very impressive. Very cool comparison @asomoza! I'd love for this to be a feature available across inpaint pipelines.

Btw, the changes for this can be fully implemented with callback_on_step_begin (I've been looking for good use cases of #6256 and this looked like one, so I gave it a shot at here).

[vladmandic]

I've been playing around with this as well and the results are very impressive. Very cool comparison @asomoza! I'd love for this to be a feature available across inpaint pipelines.

Btw, the changes for this can be fully implemented with callback_on_step_begin (I've been looking for good use cases of #6256 and this looked like one, so I gave it a shot at here).

totally off-topic here, but i've been doing color grading on latents (tint shifts, dynamic range correction, re-sharpen, etc.) in that callback. works really nice if you bind operations to specific timestep ranges.

[asomoza]

one more quick note - i like two tunable params to play with here - denoising strength as usual, but also applying relative mask brightness (-1..1), it gives so much freedom on how strong is mask being applied.

With your comment I did a quick test, this was made with just changing the brightness of the depthmap to see the effect:

prompt = "anime still"
negative_prompt = "realistic, photo"

Had to crop it because it seems that "anime" also means "open shirt" to the Juggernaut model

original	mask

Also this is what you can do with a gradient instead of a depth map:

[asomoza]

Btw, the changes for this can be fully implemented with callback_on_step_begin (I've been looking for good use cases of #6256 and this looked like one, so I gave it a shot at here).

yeah, if the diffusers team doesn't add it to the core functionality they can still mention it as a alternative.

totally off-topic here, but i've been doing color grading on latents (tint shifts, dynamic range correction, re-sharpen, etc.) in that callback. works really nice if you bind operations to specific timestep ranges.

I Intend to do that too but there is some much going on that I don't have the time to do it all, are you going to add it to SD.Next, I would really like to test it with an UI instead of code.

[vladmandic]

It's already added to SDnext. Its in "corrections" accordians and you can play with xyz grid as well.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.