feat: Add Modular Pipeline for Stable Diffusion 3 (SD3) by AlanPonnachan · Pull Request #13324 · huggingface/diffusers

AlanPonnachan · 2026-03-24T17:33:51Z

What does this PR do?

This PR introduces the modular architecture for Stable Diffusion 3 (SD3), implementing both Text-to-Image (T2I) and Image-to-Image (I2I) pipelines.

Key additions:

Added SD3ModularPipeline and SD3AutoBlocks to the dynamic modular pipeline resolver.
Migrated SD3-specific mechanics to the new BlockState
Added corresponding dummy objects and lazy-loading fallbacks.
Added TestSD3ModularPipelineFast and TestSD3Img2ImgModularPipelineFast test suites.

Related issue: #13295

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Usage Example

import torch
from IPython.display import display
from diffusers import ComponentsManager
from diffusers.modular_pipelines.stable_diffusion_3 import StableDiffusion3ModularPipeline, StableDiffusion3AutoBlocks
from diffusers.utils import load_image

from diffusers import FlowMatchEulerDiscreteScheduler, SD3Transformer2DModel, AutoencoderKL
from diffusers.guiders import ClassifierFreeGuidance
from diffusers.image_processor import VaeImageProcessor
from transformers import CLIPTokenizer, CLIPTextModelWithProjection

components = ComponentsManager()
components.enable_auto_cpu_offload(device="cuda")

# Instantiate the Modular Pipeline 
blocks = StableDiffusion3AutoBlocks()
pipeline = StableDiffusion3ModularPipeline(blocks=blocks, components_manager=components)

repo_id = "stabilityai/stable-diffusion-3-medium-diffusers"
print("Loading components...")

# Load ONLY CLIP tokenizers
tokenizer = CLIPTokenizer.from_pretrained(repo_id, subfolder="tokenizer")
tokenizer_2 = CLIPTokenizer.from_pretrained(repo_id, subfolder="tokenizer_2")

# Load diffusers components
scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
guider = ClassifierFreeGuidance.from_config({"guidance_scale": 7.0})
image_processor = VaeImageProcessor(vae_scale_factor=8, vae_latent_channels=16)

# Load ONLY CLIP text encoders
text_encoder = CLIPTextModelWithProjection.from_pretrained(repo_id, subfolder="text_encoder", torch_dtype=torch.float16)
text_encoder_2 = CLIPTextModelWithProjection.from_pretrained(repo_id, subfolder="text_encoder_2", torch_dtype=torch.float16)

# Load Transformer and VAE
transformer = SD3Transformer2DModel.from_pretrained(repo_id, subfolder="transformer", torch_dtype=torch.float16)
vae = AutoencoderKL.from_pretrained(repo_id, subfolder="vae", torch_dtype=torch.float16)

# Inject components directly into the pipeline
pipeline.update_components(
    tokenizer=tokenizer,
    tokenizer_2=tokenizer_2,
    tokenizer_3=None,    # Dropped to prevent OOM
    scheduler=scheduler,
    guider=guider,
    image_processor=image_processor,
    text_encoder=text_encoder,
    text_encoder_2=text_encoder_2,
    text_encoder_3=None, # Dropped to prevent OOM
    transformer=transformer,
    vae=vae
)

print("Components loaded successfully! Memory saved.")


# TEXT-TO-IMAGE 

prompt = "A highly detailed macro photography of a glowing bioluminescent blue butterfly resting on a vibrant red rose, dark magical forest background, cinematic lighting, 8k resolution, masterpiece"

print("Running Text-to-Image...")
t2i_output = pipeline(
    prompt=prompt,
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=torch.manual_seed(42)
)
t2i_output.images[0].save("sd3_modular_t2i.png")
print("Saved sd3_modular_t2i.png")
display(t2i_output.images[0])


# IMAGE-TO-IMAGE 

init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png").resize((1024, 1024))

prompt_i2i = "A beautiful classic impressionist oil painting of a cat looking at the camera, thick expressive brushstrokes, vibrant colors, museum masterpiece"

print("Running Image-to-Image...")
i2i_output = pipeline(
    prompt=prompt_i2i,
    image=init_image,
    strength=0.8,
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=torch.manual_seed(42)
)
i2i_output.images[0].save("sd3_modular_i2i.png")
print("Saved sd3_modular_i2i.png")
display(i2i_output.images[0])

Colab notebook: https://colab.research.google.com/drive/18_tZWIQdObq8EX0Vyd9ysGA-oACDwpf8?usp=sharing

Outputs

Text-to-Image:

Image-to-Image:

Who can review?

@sayakpaul @asomoza

tests/modular_pipelines/stable_diffusion_3/test_modular_pipeline_stable_diffusion_3.py

sayakpaul · 2026-03-25T02:24:45Z

@AlanPonnachan thanks for this PR! Could you also provide some test code and sample outputs?

sayakpaul

Thanks for getting started on this! I left some comments (majorly on the use of guidance).

src/diffusers/modular_pipelines/stable_diffusion_3/before_denoise.py

src/diffusers/modular_pipelines/stable_diffusion_3/denoise.py

src/diffusers/modular_pipelines/stable_diffusion_3/encoders.py

sayakpaul · 2026-03-28T05:27:27Z

@claude can you review this?

claude · 2026-03-28T05:27:43Z

Claude Code is working…

I'll analyze this and get back to you.

View job run

sayakpaul · 2026-03-28T05:28:02Z

@bot /style

github-actions · 2026-03-28T05:28:27Z

Style bot fixed some files and pushed the changes.

HuggingFaceDocBuilderDev · 2026-03-28T05:41:47Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

AlanPonnachan · 2026-03-29T06:18:34Z

@sayakpaul
test_modular_pipeline_stable_diffusion_3.py tests are passing.

Sample outputs you can find here: #13324 (comment)

src/diffusers/modular_pipelines/stable_diffusion_3/modular_blocks_stable_diffusion_3.py

yiyixuxu

thanks for working on this!
I left one comment

src/diffusers/modular_pipelines/stable_diffusion_3/denoise.py

- add autodocstring to assembled blocks

yiyixuxu · 2026-04-01T20:10:34Z

src/diffusers/modular_pipelines/stable_diffusion_3/modular_blocks_stable_diffusion_3.py

+logger = logging.get_logger(__name__)
+
+
+# auto_docstring


i added a doc page on this here #13382
basically you need to run

python utils/modular_auto_docstring.py --fix_and_overwrite

and to look through the generated docstring to see if all the paramters are properly defined

@yiyixuxu , I added description to most of InputParam and OutputParam and ran the above script.
I skimmed throught the docstrings once and felt right .
Let me know your thoughts!

yiyixuxu · 2026-04-01T20:11:04Z

@claude
can you do a review here?

github-actions · 2026-04-01T20:11:20Z

Claude Code is working…

I'll analyze this and get back to you.

View job run

yiyixuxu · 2026-04-02T20:40:21Z

does this work?

pipe = ModularPipeline.from("stabilityai/stable-diffusion-3-medium-diffusers")
pipe.load_components(torch_dtype=torch.float16)

@claude can you help explain what it needs to be done for this to work? based on the current state of PR

github-actions · 2026-04-02T20:40:39Z

Claude Code is working…

I'll analyze this and get back to you.

View job run

AlanPonnachan · 2026-04-03T08:20:35Z

pipe = ModularPipeline.from("stabilityai/stable-diffusion-3-medium-diffusers")
pipe.load_components(torch_dtype=torch.float16)

@yiyixuxu , I tried out this and working fine.

I ran following example on colab T4:

import torch
from IPython.display import display
from diffusers import ComponentsManager, ModularPipeline
from diffusers.utils import load_image

components = ComponentsManager()
components.enable_auto_cpu_offload(device="cuda")

repo_id = "stabilityai/stable-diffusion-3-medium-diffusers"

pipeline = ModularPipeline.from_pretrained(repo_id, components_manager=components)



components_to_load =[
    "tokenizer", "tokenizer_2", "scheduler", "guider", 
    "image_processor", "text_encoder", "text_encoder_2", 
    "transformer", "vae"
]
pipeline.load_components(components_to_load, torch_dtype=torch.float16)

pipeline.update_components(tokenizer_3=None, text_encoder_3=None)




#  TEXT-TO-IMAGE

prompt = "A highly detailed macro photography of a glowing bioluminescent blue butterfly resting on a vibrant red rose, dark magical forest background, cinematic lighting, 8k resolution, masterpiece"

print("Running Text-to-Image...")
t2i_output = pipeline(
    prompt=prompt,
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=torch.manual_seed(42)
)
t2i_output.images[0].save("sd3_modular_t2i.png")
display(t2i_output.images[0])



#  IMAGE-TO-IMAGE

init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png").resize((1024, 1024))
# Transforming the cat into a dreamy watercolor painting
prompt_i2i = "A dreamy watercolor painting of a cat sitting in a magical glowing forest, soft pastel colors, ethereal atmosphere, highly detailed, trending on artstation"

print("Running Image-to-Image...")
i2i_output = pipeline(
    prompt=prompt_i2i,
    image=init_image,
    strength=0.85, # Slightly higher strength for a stronger stylistic change
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=torch.manual_seed(100)
)
i2i_output.images[0].save("sd3_modular_i2i_watercolor.png")
display(i2i_output.images[0])

And the outputs

TEXT-TO-IMAGE

IMAGE-TO-IMAGE

AlanPonnachan added 3 commits March 23, 2026 15:53

initial architecture

f944a64

add blocks to various inits

08d14c6

styling

0a81741

AlanPonnachan commented Mar 24, 2026

View reviewed changes

tests/modular_pipelines/stable_diffusion_3/test_modular_pipeline_stable_diffusion_3.py Outdated Show resolved Hide resolved

sayakpaul requested review from asomoza and yiyixuxu March 25, 2026 02:22

sayakpaul reviewed Mar 25, 2026

View reviewed changes

AlanPonnachan added 3 commits March 26, 2026 15:09

push tiny-sd3-modular to hub and fix the tests

ba29383

rename modules

ad15c9d

guidance refactoring

02bb2af

Apply style fixes

24618de

set default height and width

27cb9f7

Merge branch 'main' into feat/sd3-modular-pipeline

651abb9

yiyixuxu reviewed Mar 30, 2026

View reviewed changes

src/diffusers/modular_pipelines/stable_diffusion_3/modular_blocks_stable_diffusion_3.py Show resolved Hide resolved

Merge branch 'main' into feat/sd3-modular-pipeline

5ad8713

yiyixuxu reviewed Mar 30, 2026

View reviewed changes

src/diffusers/modular_pipelines/stable_diffusion_3/denoise.py Outdated Show resolved Hide resolved

AlanPonnachan and others added 2 commits April 1, 2026 17:43

- skip layer refactoring

6fd0f30

- add autodocstring to assembled blocks

Merge branch 'main' into feat/sd3-modular-pipeline

08552e6

AlanPonnachan requested a review from yiyixuxu April 1, 2026 17:53

yiyixuxu reviewed Apr 1, 2026

View reviewed changes

add description and run autostring script

956f363

AlanPonnachan and others added 4 commits April 2, 2026 17:19

styling

e375cfa

Merge branch 'main' into feat/sd3-modular-pipeline

63ddbbd

add descriptions for outputparams and styling

8626d27

Merge branch 'main' into feat/sd3-modular-pipeline

ea07777

Conversation

AlanPonnachan commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Usage Example

Outputs

Text-to-Image:

Image-to-Image:

Who can review?

Uh oh!

Uh oh!

sayakpaul commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul commented Mar 28, 2026

Uh oh!

claude bot commented Mar 28, 2026

Uh oh!

sayakpaul commented Mar 28, 2026

Uh oh!

github-actions bot commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Mar 28, 2026

Uh oh!

AlanPonnachan commented Mar 29, 2026

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

AlanPonnachan Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Apr 1, 2026

Uh oh!

github-actions bot commented Apr 1, 2026

Uh oh!

yiyixuxu commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

AlanPonnachan commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TEXT-TO-IMAGE

IMAGE-TO-IMAGE

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AlanPonnachan commented Mar 24, 2026 •

edited

Loading

sayakpaul commented Mar 25, 2026 •

edited

Loading

github-actions bot commented Mar 28, 2026 •

edited

Loading

AlanPonnachan commented Apr 3, 2026 •

edited

Loading