[Bluear7878]

What does this PR do?

Fixes #47920

CLIPPreTrainedModel._init_weights initializes the parameters of child modules by name —
module.q_proj.weight, module.fc1.weight, module.text_projection.weight, and so on. A quantized
checkpoint replaces weight on those children with packed tensors (weight_packed for
compressed-tensors, qweight for GPTQ/AWQ), so loading one raises

AttributeError: 'Linear' object has no attribute 'weight'

from the initialization pass that runs at the end of loading: _finalize_model_loading →
_initialize_missing_keys → initialize_weights → smart_apply → _init_weights
(modeling_utils.py:4517, :2476-2491). For a packed format the packed layout is installed before
the weights are loaded, by transformers itself —
CompressedTensorsHfQuantizer._process_model_before_weight_loading calls
self.compressor.compress_model(model) (quantizers/quantizer_compressed_tensors.py:117-121).

A container such as CLIPAttention owns no parameter of its own, so _initialize_weights' skip
conditions never apply to it for a class living in transformers.*, and it always reaches
_init_weights.

This PR routes every child-weight initialization in CLIPPreTrainedModel._init_weights through a
small helper that skips a child with no weight, which is the guard the generic
PreTrainedModel._init_weights already applies to nn.Linear/nn.Conv* (modeling_utils.py:2392)
and to the norms (:2418-2421, "Norms can exist without weights"). It is the same shape as the
merged #38013, which fixed this class of failure in modeling_qwen2_5_omni.py for AutoAWQ
checkpoints.

MetaClip2PreTrainedModel inherits this method through modular and the converter inlines the body,
so models/metaclip_2/modeling_metaclip_2.py is regenerated in the same commit
(python utils/modular_model_converter.py --files_to_parse src/transformers/models/metaclip_2/modular_metaclip_2.py);
MetaCLIP-2 gets the fix with it.

Initialization of a non-quantized model is unchanged: the seeded state_dict of a tiny CLIPModel
hashes to the same SHA-256 before and after the patch (a6808d27…). The change also removes a
pre-existing crash for CLIPForImageClassification with num_labels == 0, where classifier is an
nn.Identity.

Verification

# the new regression test
pytest tests/models/clip/test_modeling_clip.py -k init_weights_with_quantized
  -> 1 passed          (with the patch)
  -> 1 failed, AttributeError: 'Linear' object has no attribute 'weight'   (with the patch reverted)

# the test files utils/tests_fetcher.py selects for this diff
pytest tests/models/clip/test_modeling_clip.py tests/models/metaclip_2/test_modeling_metaclip_2.py \
       tests/models/sam3/test_modeling_sam3.py tests/models/sam3_video/test_modeling_sam3_video.py \
       tests/models/vision_text_dual_encoder/test_modeling_vision_text_dual_encoder.py
  -> 869 passed, 960 skipped, 6 xfailed, 8 failed in 6m18s

# repo checks
python utils/checkers.py ruff_check, ruff_format, init_isort, sort_auto_mappings, modeling_structure
  -> all OK (ruff over the whole tree)
python utils/checkers.py <all REPO_CONSISTENCY_CHECKERS>
  -> All 17 checks passed

The 8 failures are all test_multi_gpu_data_parallel_forward (4 in the CLIP file, 4 in the MetaCLIP-2
file). They are pre-existing on this machine: the same 8 fail on the parent commit, and nothing else in
those five files fails.

The types checker fails identically on the parent commit with the ty version I have installed, so
it is not affected by this diff.

Scope

The same pattern — _init_weights dereferencing a child module's .weight/.bias — is present in 83
of 488 modeling_*.py files on main, and I measured the identical AttributeError for T5,
Mamba2, LLaVA (CLIP vision tower) and Gemma3 (SigLIP vision tower). The issue linked above has
the full analysis, including why marking the quantized modules _is_hf_initialized (as the
bitsandbytes/torchao/quanto/mxfp4/sinq integrations do) does not fix it, and why the 4.x
"skip a fully loaded subtree" rule does not either. I kept this PR to CLIP; happy to extend the guard
to the other families in whatever batching you prefer.

Code Agent Policy

• (First-time contributors only): I confirm that this PR description and code is not written by an LLM or code agent

Not applicable — I am not a first-time contributor (#46277, #47940). Per the Agentic contributions
section of CONTRIBUTING.md: this change was prepared with AI assistance (Claude Code). I have
reviewed every changed line, the analysis and every number above come from runs against the installed
packages, the test commands and results are quoted above, the coordination link is #47920, and I am
not aware of another open PR touching CLIP's _init_weights. I opened this alongside the issue so the
diff is concrete rather than hypothetical -- if you would rather settle the scope there first (one
model vs. the 83 files that share the pattern), say so and I will hold this PR until then.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case. — see the linked issue
• Did you make sure to update the documentation with your changes according to the guidelines? — no documented behavior changes
• Did you write any new necessary tests?
  tests/models/clip/test_modeling_clip.py::CLIPModelTest::test_init_weights_with_quantized_children

Who can review?

@Cyrilvallez (the failure is in the from_pretrained initialization pass), @molbap (CLIP)

[Rocketknight1]

I took a look and this does make sense. Most models don't use this approach but CLIP does, and it does break when combined with quantization. The fix is clean enough too, so I'm happy to merge it!

[Rocketknight1]

Also @Bluear7878 although this pattern is quite common, most models use the base method for init weights, which doesn't trigger this issue. So the only models where this init pattern is problematic are the ones that override the weight init method and access child weights like this.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: clip, metaclip_2

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 31602240410:2
Result: success | Jobs: 2 | Tests: 9 | Failures: 2 | Duration: 1m 0s