Skip to content

[SDXL DreamBooth LoRA] add support for text encoder fine-tuning #4097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Jul 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
65a5c45
Allow low precision sd xl
patrickvonplaten Jul 13, 2023
43f842c
finish
patrickvonplaten Jul 13, 2023
7171d42
finish
patrickvonplaten Jul 13, 2023
97f69a7
Merge branch 'main' into allow_low_precision_vae_sd_xl
patrickvonplaten Jul 13, 2023
9337535
feat: initial draft for supporting text encoder lora finetuning for S…
sayakpaul Jul 14, 2023
69e9bfa
Merge remote-tracking branch 'origin/allow_low_precision_vae_sd_xl' i…
sayakpaul Jul 14, 2023
b487dfc
fix: variable assignments.
sayakpaul Jul 14, 2023
c51e559
add: autocast block.
sayakpaul Jul 14, 2023
9d23e30
add debugging
sayakpaul Jul 14, 2023
ea285db
vae dtype hell
sayakpaul Jul 14, 2023
b0ed1b6
fix: vae dtype hell.
sayakpaul Jul 14, 2023
20a9186
fix: vae dtype hell 3.
sayakpaul Jul 14, 2023
9d3e606
clean up
sayakpaul Jul 14, 2023
35b8dae
lora text encoder loader.
sayakpaul Jul 14, 2023
9c305b1
fix: unwrapping models.
sayakpaul Jul 14, 2023
4afb793
add: tests.
sayakpaul Jul 14, 2023
42fb433
docs.
sayakpaul Jul 14, 2023
0f95887
handle unexpected keys.
sayakpaul Jul 14, 2023
ede8ca2
fix vae dtype in the final inference.
sayakpaul Jul 14, 2023
34b536c
fix scope problem.
sayakpaul Jul 14, 2023
ad26174
fix: save_model_card args.
sayakpaul Jul 14, 2023
c5a95d6
initialize: prefix to None.
sayakpaul Jul 14, 2023
63f62b4
fix: dtype issues.
sayakpaul Jul 14, 2023
2d815d2
apply gixes.
sayakpaul Jul 14, 2023
0df9af2
Merge branch 'main' into feat/sdxl-dreambooth-returns
sayakpaul Jul 14, 2023
3bb3d4f
debgging.
sayakpaul Jul 14, 2023
6b496d5
debugging
sayakpaul Jul 14, 2023
d4a68a9
debugging
sayakpaul Jul 14, 2023
b70b003
debugging
sayakpaul Jul 14, 2023
fe9ca14
debugging
sayakpaul Jul 14, 2023
3d968bb
debugging
sayakpaul Jul 14, 2023
bcac032
add: fast tests.
sayakpaul Jul 18, 2023
ff3f27f
pre-tokenize.
sayakpaul Jul 18, 2023
0c33566
address: will's comments.
sayakpaul Jul 18, 2023
43d7046
fix: loader and tests.
sayakpaul Jul 18, 2023
8d5d5b1
fix: dataloader.
sayakpaul Jul 18, 2023
0d77b53
simplify dataloader.
sayakpaul Jul 21, 2023
c80915e
Merge branch 'main' into feat/sdxl-dreambooth-returns
sayakpaul Jul 21, 2023
91b0c3a
length.
sayakpaul Jul 21, 2023
52eef75
simplification.
sayakpaul Jul 21, 2023
ef501c8
make style && make quality
sayakpaul Jul 21, 2023
07a45c8
simplify state_dict munging
sayakpaul Jul 21, 2023
6ca45f3
fix: tests.
sayakpaul Jul 21, 2023
5f4e089
fix: state_dict packing.
sayakpaul Jul 21, 2023
989e54d
Apply suggestions from code review
sayakpaul Jul 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions examples/dreambooth/README_sdxl.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,17 @@ Here's a side-by-side comparison of the with and without Refiner pipeline output
|---|---|
| ![](https://huggingface.co/datasets/diffusers/docs-images/resolve/main/sd_xl/sks_dog.png) | ![](https://huggingface.co/datasets/diffusers/docs-images/resolve/main/sd_xl/refined_sks_dog.png) |

### Training with text encoder(s)

Alongside the UNet, LoRA fine-tuning of the text encoders is also supported. To do so, just specify `--train_text_encoder` while launching training. Please keep the following points in mind:

* SDXL has two text encoders. So, we fine-tune both using LoRA.
* When not fine-tuning the text encoders, we ALWAYS precompute the text embeddings to save memory.

### Specifying a better VAE

SDXL's VAE is known to suffer from numerical instability issues. This is why we also expose a CLI argument namely `--pretrained_vae_model_name_or_path` that lets you specify the location of a better VAE (such as [this one](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix)).

## Notes

In our experiments we found that SDXL yields very good initial results using the default settings of the script. We didn't explore further hyper-parameter tuning experiments, but we do encourage the community to explore this avenue further and share their results with us 🤗
Expand Down
Loading