|
| 1 | +# Consistency Models |
| 2 | + |
| 3 | +Consistency Models were proposed in [Consistency Models](https://arxiv.org/abs/2303.01469) by Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. |
| 4 | + |
| 5 | +The abstract of the [paper](https://arxiv.org/pdf/2303.01469.pdf) is as follows: |
| 6 | + |
| 7 | +*Diffusion models have significantly advanced the fields of image, audio, and video generation, but they depend on an iterative sampling process that causes slow generation. To overcome this limitation, we propose consistency models, a new family of models that generate high quality samples by directly mapping noise to data. They support fast one-step generation by design, while still allowing multistep sampling to trade compute for sample quality. They also support zero-shot data editing, such as image inpainting, colorization, and super-resolution, without requiring explicit training on these tasks. Consistency models can be trained either by distilling pre-trained diffusion models, or as standalone generative models altogether. Through extensive experiments, we demonstrate that they outperform existing distillation techniques for diffusion models in one- and few-step sampling, achieving the new state-of-the-art FID of 3.55 on CIFAR-10 and 6.20 on ImageNet 64x64 for one-step generation. When trained in isolation, consistency models become a new family of generative models that can outperform existing one-step, non-adversarial generative models on standard benchmarks such as CIFAR-10, ImageNet 64x64 and LSUN 256x256. * |
| 8 | + |
| 9 | +Resources: |
| 10 | + |
| 11 | +* [Paper](https://arxiv.org/abs/2303.01469) |
| 12 | +* [Original Code](https://github.com/openai/consistency_models) |
| 13 | + |
| 14 | +Available Checkpoints are: |
| 15 | +- *cd_imagenet64_l2 (64x64 resolution)* [openai/consistency-model-pipelines](https://huggingface.co/openai/consistency-model-pipelines) |
| 16 | +- *cd_imagenet64_lpips (64x64 resolution)* [openai/diffusers-cd_imagenet64_lpips](https://huggingface.co/openai/diffusers-cd_imagenet64_lpips) |
| 17 | +- *ct_imagenet64 (64x64 resolution)* [openai/diffusers-ct_imagenet64](https://huggingface.co/openai/diffusers-ct_imagenet64) |
| 18 | +- *cd_bedroom256_l2 (256x256 resolution)* [openai/diffusers-cd_bedroom256_l2](https://huggingface.co/openai/diffusers-cd_bedroom256_l2) |
| 19 | +- *cd_bedroom256_lpips (256x256 resolution)* [openai/diffusers-cd_bedroom256_lpips](https://huggingface.co/openai/diffusers-cd_bedroom256_lpips) |
| 20 | +- *ct_bedroom256 (256x256 resolution)* [openai/diffusers-ct_bedroom256](https://huggingface.co/openai/diffusers-ct_bedroom256) |
| 21 | +- *cd_cat256_l2 (256x256 resolution)* [openai/diffusers-cd_cat256_l2](https://huggingface.co/openai/diffusers-cd_cat256_l2) |
| 22 | +- *cd_cat256_lpips (256x256 resolution)* [openai/diffusers-cd_cat256_lpips](https://huggingface.co/openai/diffusers-cd_cat256_lpips) |
| 23 | +- *ct_cat256 (256x256 resolution)* [openai/diffusers-ct_cat256](https://huggingface.co/openai/diffusers-ct_cat256) |
| 24 | + |
| 25 | +## Available Pipelines |
| 26 | + |
| 27 | +| Pipeline | Tasks | Demo | Colab | |
| 28 | +|:---:|:---:|:---:|:---:| |
| 29 | +| [ConsistencyModelPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_consistency_models.py) | *Unconditional Image Generation* | | | |
| 30 | + |
| 31 | +This pipeline was contributed by our community members [dg845](https://github.com/dg845) and [ayushtues](https://huggingface.co/ayushtues) :heart: |
| 32 | + |
| 33 | +## Usage Example |
| 34 | + |
| 35 | +```python |
| 36 | +import torch |
| 37 | + |
| 38 | +from diffusers import ConsistencyModelPipeline |
| 39 | + |
| 40 | +device = "cuda" |
| 41 | +# Load the cd_imagenet64_l2 checkpoint. |
| 42 | +model_id_or_path = "openai/diffusers-cd_imagenet64_l2" |
| 43 | +pipe = ConsistencyModelPipeline.from_pretrained(model_id_or_path, torch_dtype=torch.float16) |
| 44 | +pipe.to(device) |
| 45 | + |
| 46 | +# Onestep Sampling |
| 47 | +image = pipe(num_inference_steps=1).images[0] |
| 48 | +image.save("consistency_model_onestep_sample.png") |
| 49 | + |
| 50 | +# Onestep sampling, class-conditional image generation |
| 51 | +# ImageNet-64 class label 145 corresponds to king penguins |
| 52 | +image = pipe(num_inference_steps=1, class_labels=145).images[0] |
| 53 | +image.save("consistency_model_onestep_sample_penguin.png") |
| 54 | + |
| 55 | +# Multistep sampling, class-conditional image generation |
| 56 | +# Timesteps can be explicitly specified; the particular timesteps below are from the original Github repo. |
| 57 | +# https://github.com/openai/consistency_models/blob/main/scripts/launch.sh#L77 |
| 58 | +image = pipe(timesteps=[22, 0], class_labels=145).images[0] |
| 59 | +image.save("consistency_model_multistep_sample_penguin.png") |
| 60 | +``` |
| 61 | + |
| 62 | +For an additional speed-up, one can also make use of `torch.compile`. Multiple images can be generated in <1 second as follows: |
| 63 | + |
| 64 | +```py |
| 65 | +import torch |
| 66 | +from diffusers import ConsistencyModelPipeline |
| 67 | + |
| 68 | +device = "cuda" |
| 69 | +# Load the cd_bedroom256_lpips checkpoint. |
| 70 | +model_id_or_path = "openai/diffusers-cd_bedroom256_lpips" |
| 71 | +pipe = ConsistencyModelPipeline.from_pretrained(model_id_or_path, torch_dtype=torch.float16) |
| 72 | +pipe.to(device) |
| 73 | + |
| 74 | +pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True) |
| 75 | + |
| 76 | +# Multistep sampling |
| 77 | +# Timesteps can be explicitly specified; the particular timesteps below are from the original Github repo: |
| 78 | +# https://github.com/openai/consistency_models/blob/main/scripts/launch.sh#L83 |
| 79 | +for _ in range(10): |
| 80 | + image = pipe(timesteps=[17, 0]).images[0] |
| 81 | + image.show() |
| 82 | +``` |
| 83 | + |
| 84 | +## ConsistencyModelPipeline |
| 85 | +[[autodoc]] ConsistencyModelPipeline |
| 86 | + - all |
| 87 | + - __call__ |
0 commit comments