[Modular] loader related

[yiyixuxu]

WIP
a few loading related refactor
will also add a test suit along this PR

refactor1: tag model with _diffusers_load_id inside AutoModel.from_pretrained

AutoModel now tags a _diffusers_load_id on loaded models, this is to make it easier to serialize ModolarPipeline into a modular_model_index.json with info such as pretrained_model_name_or_path, subfolder, variant, revision

running this under main would get a warn

ModularPipeline.update_components: unet has no valid _diffusers_load_id. This will result in empty loading spec, use ComponentSpec.load() for proper specs

but it would work without an issue under this PR

from diffusers import ModularPipeline, AutoModel
pipe = ModularPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
auto_model = AutoModel.from_pretrained("RunDiffusion/Juggernaut-XL-v9", subfolder="unet", variant="fp16")
pipe.update_components(unet=auto_model)
print(pipe) 

refactor 2: more intuitive (I hope!) defaults behavior for load_components()

load_components() now skips components that are already loaded or don't have a valid pretrained_model_name_or_path in their spec. This avoids unnecessary reloading and confusing error messages for optional components. See inline comments for details.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

with this change, now, by default with pipeline.load_components():

1. we don't load components that's already loaded in pipeline
2. we don't load the components that do not have valid pretrained_model_name_or_path

for (1), currently, under the main, if we run script below, we will load the text_encoder, add it to the pipeline with update_components, and then reload again when you run load_components() - which is a bit unintuitive I think

in this PR, it will skip text_encoder instead of loading all components

pipe = ModularPipeline.from_pretrained()
text_encoder = AutoModel.from_pretrained(...)
pipe.update_components(text_encoder=text_encoder )
pipe.load_components(torch_dtype=torch.bfloat16)

for (2), in main, if you run this

from diffusers import ModularPipeline
pipe = ModularPipeline.from_pretrained("Qwen/Qwen-Image")
pipe.load_components(torch_dtype=torch.bfloat16)

you would get a confusing message like this

Failed to create component controlnet:
- Component spec: ComponentSpec(name='controlnet', type_hint=<class 'diffusers.models.controlnets.controlnet_qwenimage.QwenImageControlNetModel'>, description=None, config=None, pretrained_model_name_or_path=None, subfolder='', variant=None, revision=None, default_creation_method='from_pretrained', repo=None)
- load() called with kwargs: {'torch_dtype': torch.bfloat16}
If this component is not required for your workflow you can safely ignore this message.

Traceback:
Traceback (most recent call last):
  File "/fsx/yiyi/diffusers/src/diffusers/modular_pipelines/modular_pipeline.py", line 2173, in load_components
    components_to_register[name] = spec.load(**component_load_kwargs)
  File "/fsx/yiyi/diffusers/src/diffusers/modular_pipelines/modular_pipeline_utils.py", line 279, in load
    raise ValueError(
ValueError: `pretrained_model_name_or_path` info is required when using `load` method (you can directly set it in `pretrained_model_name_or_path` field of the ComponentSpec or pass it as an argument)

this is because controlnet is a component in qwen auto pipeline, but not included in Qwen/Qwen-Image; in this PR, we just skip it by default and not sending any message - I think it is more of an expected behavior

[yiyixuxu]

@DN6 @sayakpaul

ready for an initial review! let me know if you have other suggestions for making the loading behavior more intuitive/user-friendly

once we are happy with it, we want to start building out a test suite and updating the docs.

[DN6]

LGTM 👍🏽

[sayakpaul]

I am guessing pretrained_model_name_or_path will always be set for a component that's expected?

[yiyixuxu]

It is up to the user.

We work with modular_model_index.json like this: https://huggingface.co/sayakpaul/new-template/blob/main/modular_model_index.json

Unlike model_index.json, you can update it manually to make the pipeline load from different repos.

This one was automatically converted from a regular Qwen Pipeline:

ModularPipeline.from_pretrained("Qwen/Qwen-Image").save_pretrained()

So it has all the pretrained_model_name_or_path info for all the components in "Qwen/Qwen-Image" but not the ControlNet.

Custom ones can be all empty to start with:

MyCustomBlock.init_pipeline().save_pretrained()

Another example: https://huggingface.co/diffusers/flux2-bnb-4bit-modular/blob/main/modular_model_index.json#L49

Here we updated the pretrained_model_name_or_path for text_encoder/transformer to load from quantized checkpoints instead.

So it's very flexible - and I think it's expected behavior that if pretrained_model_name_or_path is not set, we just skip loading that component (vs throwing an error).

[sayakpaul]

That makes sense, thanks for explaining. I was missing the point on the latter.