Skip to content

Resume Function in Dreambooth Lora Training is Broken since 0.19.0 #4936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dreamraster opened this issue Sep 7, 2023 · 5 comments
Closed
Labels
bug Something isn't working stale Issues that haven't received updates

Comments

@dreamraster
Copy link

Describe the bug

Resume functionality in train_dreambooth_lora.py example seems to be completely broken since 0.19.0 and narrowed down to #3778
Reverting 0.18.2 style temp_pipeline does seem to fix textencoder resume but not unet.

Reproduction

The Repro runs training with 2 resumes for the 0.18.2 and 0.19.

Add a few images to input folder and run the repro.bat

For version 0.18.2
Both with and without train_text_encoder, a new pytorch_lora_weights.bin is created properly as resume works just fine.

For version 0.19.3
Both with and without train_text_encoder, a new pytorch_lora_weights.bin is not created on resume steps and hence pytorch_lora_weights.bin is always the same.

Resume functionality seems to be completely broken since 0.19.0 and narrowed down to #3778
Reverting 0.18.2 style temp_pipeline does seem to fix textencoder resume but not unet.

Please remove txt extension of the files

train_dreambooth_lora_0.18.2_repro.py.txt
train_dreambooth_lora_0.19.0_repro.py.txt
repro.bat.txt

Logs

No response

System Info

  • diffusers version: 0.19.0
  • Platform: Windows-10-10.0.22621-SP0
  • Python version: 3.10.4
  • PyTorch version (GPU?): 2.0.1+cu118 (True)
  • Huggingface_hub version: 0.16.4
  • Transformers version: 4.33.0
  • Accelerate version: 0.22.0
  • xFormers version: not installed
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: no

Who can help?

@williamberman, @sayakpaul, @yiyixuxu

@dreamraster dreamraster added the bug Something isn't working label Sep 7, 2023
@sayakpaul
Copy link
Member

You should clone the latest main of diffusers and take it from there.

@dreamraster
Copy link
Author

@sayakpaul
Just fyi: Issue persist in v0.20.2 as well. Will try mainline. Any inputs or suggestions or known issues regarding this ?

@sayakpaul
Copy link
Member

@williamberman

@dreamraster
Copy link
Author

Did a bisection and it seems PR #4765 fixes it.
So versions >= 0.19.0 < 0.21.0 is basically broken for checkpoint resume. Any pointers on what could have caused this please ?

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label Oct 18, 2023
@github-actions github-actions bot closed this as completed Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale Issues that haven't received updates
Projects
None yet
Development

No branches or pull requests

2 participants