Stable Cascade Controlnet training

[geroldmeisinger]

Is your feature request related to a problem? Please describe.
Last year I wrote a long article on how to train controlnets using diffusers, and trained two controlnets using diffusers. A few months ago Stable Cascade was released which requires less resources than SDXL and also should make controlnet training for high quality diffusion models more viable. I tried to run the official training script (batchsize=4, bf16) but ran into OOM. I'd hope that a diffusers implementation would provide more optimization and lower VRAM requirements.

Describe the solution you'd like.
Please provide a examples/controlnet/train_controlnet_stablecascade.py

Describe alternatives you've considered.
The official training script => OOM

Additional context.
A training script for the Würstchen architecture was already anticipated a long time ago (see #5071).

Pretty please!

[sayakpaul]

Sorry, we won't have the bandwidth to work on that script right now. I will leave it open to the community in case someone from the community wants to pick it up.

[sayakpaul]

Cc: @kashif as well since he worked on the SD Cascade scripts.

[Bhavay-2001]

Hi @sayakpaul, could you pls tell how may I start working on this? Like how may I proceed in doing such tasks? Any help will be appreciated.
Thanks

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Closing due to inactivity.