You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Resume functionality in train_dreambooth_lora.py example seems to be completely broken since 0.19.0 and narrowed down to #3778
Reverting 0.18.2 style temp_pipeline does seem to fix textencoder resume but not unet.
Reproduction
The Repro runs training with 2 resumes for the 0.18.2 and 0.19.
Add a few images to input folder and run the repro.bat
For version 0.18.2
Both with and without train_text_encoder, a new pytorch_lora_weights.bin is created properly as resume works just fine.
For version 0.19.3
Both with and without train_text_encoder, a new pytorch_lora_weights.bin is not created on resume steps and hence pytorch_lora_weights.bin is always the same.
Resume functionality seems to be completely broken since 0.19.0 and narrowed down to #3778
Reverting 0.18.2 style temp_pipeline does seem to fix textencoder resume but not unet.
Did a bisection and it seems PR #4765 fixes it.
So versions >= 0.19.0 < 0.21.0 is basically broken for checkpoint resume. Any pointers on what could have caused this please ?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Describe the bug
Resume functionality in train_dreambooth_lora.py example seems to be completely broken since 0.19.0 and narrowed down to #3778
Reverting 0.18.2 style temp_pipeline does seem to fix textencoder resume but not unet.
Reproduction
The Repro runs training with 2 resumes for the 0.18.2 and 0.19.
Add a few images to input folder and run the repro.bat
For version 0.18.2
Both with and without train_text_encoder, a new pytorch_lora_weights.bin is created properly as resume works just fine.
For version 0.19.3
Both with and without train_text_encoder, a new pytorch_lora_weights.bin is not created on resume steps and hence pytorch_lora_weights.bin is always the same.
Resume functionality seems to be completely broken since 0.19.0 and narrowed down to #3778
Reverting 0.18.2 style temp_pipeline does seem to fix textencoder resume but not unet.
Please remove txt extension of the files
train_dreambooth_lora_0.18.2_repro.py.txt
train_dreambooth_lora_0.19.0_repro.py.txt
repro.bat.txt
Logs
No response
System Info
diffusers
version: 0.19.0Who can help?
@williamberman, @sayakpaul, @yiyixuxu
The text was updated successfully, but these errors were encountered: