Fix torch_dtype in Kolors text encoder with transformers v4.49

[hlky]

What does this PR do?

Tests for Kolors are failing. Tracked the issue to transformers version update. The test model's config contains torch_dtype as a string, in turn dtype is passed as str to torch.empty, seems torch_dtype was previously converted to a torch.dtype or ignored.

TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=str, device=str), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)

Generally torch_dtype would be passed to Kolors' pipelines from_pretrained or ChatGLMModel if creating it separately, so this should be ok for end users.

Edit: some tests still failing

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[sayakpaul]

The test model's config contains torch_dtype as a string, in turn dtype is passed as str to torch.empty, seems torch_dtype was previously converted to a torch.dtype or ignored.

I think transformers has a way to map that to the correct dtype:
https://github.com/huggingface/transformers/blob/e6cc410d5b830e280cdc5097cb6ce6ea6a943e5e/src/transformers/modeling_utils.py#L4081

[hlky]

Yeah, I haven't found the exact change but it was working recently on 4.48.3

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Let's maybe also update the transformers folks about this.

[hlky]

Failing test is now just a temporary Hub issue.

To re-summarize the changes we have

• Passed torch_dtype to ChatGLMModel in tests - ChatGLMModel's from_pretrained is from transformers
• Set torch.float32 as the default for torch_dtype and added a warning if the passed torch_dtype is not a torch.dtype, this is for pipeline-level tests, we're not passing torch_dtype to some so ChatGLMModel doesn't get torch_dtype which causes the issue from latest transformers.

[hlky]

Marked as draft again, waiting for huggingface/transformers#36262

[DN6]

@hlky I think this can be reopened and merged. Change is safe to make.