Training example for StableUnCLIP

[haofanwang]

The pipeline_stable_unclip_img2img.py is great, but how do I finetune the model just as done in train_text_to_image.py. For now, I achieve it on my own, but the loss doesn't decrease as expected. There seems to be of some bugs. @patil-suraj Can you PR a training example for it?

[patrickvonplaten]

Hmm yeah we haven't looked too much into fine-tuning here to be honest and I'm not sure if we find the time to do so.

[haofanwang]

It would be very appreciated if pseudocode for the core part (how the image embedding is added in the forward process) can be provided. Then I can implement the full training code. @patrickvonplaten

[eeyrw]

I am quite curious about training cascade super resolution module of unCLIP implementation. The current community solution is use SD with CLIP image embedding version to generate image which is not good.

[haofanwang]

Which community implementation do you use? @eeyrw

[eeyrw]

This one:
#1808
https://huggingface.co/docs/diffusers/v0.14.0/en/api/pipelines/stable_diffusion/image_variation#diffusers.StableDiffusionImage
But I notice https://huggingface.co/docs/diffusers/api/pipelines/stable_unclip. It seems new model based on SD2.1?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[00why00]

Same issue. Have you solved it? @haofanwang
How the image embedding is added in the forward process

[mynameischaos]

Same issue. Have you solved it? @haofanwang How the image embedding is added in the forward process

Same issue