GGUF remove de-quantizing and converting support, like comfyui

[zwukong]

GGUF now needs compile in every running, very slow. Can you use the similar codes like comfyui-GGUF to help our low-vram users. GGUF is very important for us 🤗
https://github.com/city96/ComfyUI-GGUF

[DN6]

Hi @zwukong GGUF checkpoints should work with torch compile if you're using torch>=2.7.1. e.g

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel, GGUFQuantizationConfig

model_id = "black-forest-labs/FLUX.1-dev"
ckpt_path = (
    "https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q2_K.gguf"
)
transformer = FluxTransformer2DModel.from_single_file(
    ckpt_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)
transformer = torch.compile(transformer, fullgraph=True)

pipe = FluxPipeline.from_pretrained(
    model_id, transformer=transformer, torch_dtype=torch.bfloat16
)
pipe.to("cuda")
images = pipe("A cat holding a sign that says hello").images[0]
images.save("flux-gguf-compile.png")

[zwukong]

Thanks for your reply, what i want is no compile. when gguf runs ,always compile even if you are not using torch.compile

[DN6]

Sorry @zwukong I'm not sure I follow. Are you saying running GGUF without torch compile is slow? Do you have a script or code example I can use to reproduce your issue?

[zwukong]

Your script without compile. run multi times then you will know how slow it is. always compiling. Try comfyui-GGUF,then you will know the difference. How fast comfyui version is

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel, GGUFQuantizationConfig

model_id = "black-forest-labs/FLUX.1-dev"
ckpt_path = (
    "https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q2_K.gguf"
)
transformer = FluxTransformer2DModel.from_single_file(
    ckpt_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)

pipe = FluxPipeline.from_pretrained(
    model_id, transformer=transformer, torch_dtype=torch.bfloat16
)
pipe.to("cuda")
images = pipe("A cat holding a sign that says hello").images[0]
images.save("flux-gguf-compile.png")

[asomoza]

@zwukong I think you're mixing concepts here, maybe there's a problem in translation? GGUF models don't compile anything when they run with diffusers, it's just used as a container with lower precision so the VRAM consumption is lower, don't know why are you so sure about your assumption.

Also, are you in windows? and how much VRAM do you have? If you don't provide more information is hard to help you.

I tested both with a 5090 on linux and both are mostly the same:

ComfyUI

Diffusers

the difference is mostly because ComfyUI does all the loading before the sampler node and in diffusers the models get loaded on the first step of the denoise loop.

This changes completely when using a more lower end hardware, on a mobile 4090 with 16GB of VRAM with also RAM and CPU with lower specs, I get this:

ComfyUI

Diffusers

The difference is bigger because I have the nvidia thing that makes everything slower instead of hitting a OOM.

When hitting that slow of a generation, yeah, there's a difference we can perceive between both, but for sure, it is not a compilation as you're stating, probably the difference is because comfyui is mitigating the VRAM overflow with some kind of optimization.

So please, if you want help, post more details.

[zwukong]

Yes ,windows. In my logs I remember there is another line, de-quantizing in each running and very slow

[zwukong]

Thanks for your contribution，but i don't think linux do not need that de-quantizing period, however comfyui do,very fast

[asomoza]

but i don't think linux do not need that de-quantizing period

you sound very prepotent with this line, I will assume this is just a translation issue but I won't answer again if you keep doing it, nowadays AI are very good at translating and they don't make this kind of mistake unless you're using the same kind of tone in your own language. After re reading it I think you were trying to say "period of time", meaning like a "step", so it is probably a translation issue.

I'm guessing what you're talking about because you still don't provide any more information, no reproduction code, no environment information and no system specs.

Converting is necessary because you're using a format that's not the diffusers one, so if we don't convert the weights, it's impossible to run it with diffusers. Same as dequantizing, as I told you before, GGUF is just a container.

Again, you're not providing a reproduction code, so I'm guessing, but this only happens once when you load the model, after that you can do inference multiple times without the need of it, I'm guessing you're running the whole script multiple times with the loading including which is wrong and not how you should do it, comfyui also doesn't do this, you just load the model one time.

Also I almost don't see the progress bar of the converting and dequantizing because it's really fast on any modern CPU (I see 0 seconds), so I'm also guessing you're using a very old machine to do this, 25 minutes for this step seems very wrong.

Finally, as I demonstrated, the difference on the inference with comfyui and diffusers is minimal, and also the loading is the same for me.

About if there's a real issue with the GGUF dequantizing, I'll leave that answer to @DN6 but if you don't provide more detailed information and a more proper tone and language, don't expect any more answers.

[zwukong]

Thanks for you reply. I know some python ,so i said linux must need de-quantizing period too. My cpu is old i7 8700, but i don't think cpu is an bottleneck. Gpu is 12G 4070, vram not that much, but for me it is enough. Why i open an issue is that i found that de-quantizing period very slow, comfyui need de-quantizing too, but a lot more faster than diffuser, and it can also cache the node, while diffuser version needs dequantizing in every running.

what we all want is to make the community much better, nothing else i think.

[asomoza]

thanks for the more detailed answer, but again, we need something to reproduce your issue, as I said 25 mins for that step is really wrong, and that step is done in the CPU not the GPU, so it matters a lot.

So for example, with this code:

model_id = "black-forest-labs/FLUX.1-dev"
ckpt_path = "C:/projects/python/ComfyUI/models/diffusion_models/flux1-dev-Q2_K.gguf"

transformer = FluxTransformer2DModel.from_single_file(
    ckpt_path, quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16), torch_dtype=torch.bfloat16
)

prompt_encode_pipe = FluxPipeline.from_pretrained(
    model_id,
    transformer=transformer,
    vae=None,
    torch_dtype=torch.bfloat16,
)

the GPU doesn't get any kind of workload, and I see this:

I can't even see the converting and de-quantizing bar because it is too fast, this is with a i9-14900HX (laptop), sadly I don't have a slower CPU at hand but yours is like 9 years old system, not sure if we can test and confirm that diffusers works on that kind of machines.

I can help you if you want to use diffusers models with quantization though, but you seem to like GGUF and ComfyUI so that's why I didn't propose it, diffusers works a lot better with bitsandbytes or sdnq with a group offloading combination.

[asomoza]

and it can also cache the node, while diffuser version needs dequantizing in every running.

this is something you're doing wrong, you don't have to load the model on every run, you should build a GUI or console app that runs in a loop, you can load the model one time and then do multiple inferences with the same model loaded. Diffusers is a library for developers so you have to build the whole system, doing a simple gradio app should be easy enough that you can just ask an LLM to do it for you.

[zwukong]

Thanks for you advices , i know bitsandbytes and sdnq. Only when comfyui do not have some supports i will looks for bitsandbytes and sdnq. They are very useful for small vrams like me. But sdnq not that good i think. I tried ltx2 , very bad , although some image models like z work fine. Almost all models use diffuser as their first choice, but when it comes to ecosystem ,comfyui more better i think.

Another very important thing needed to mention is VRAM managent, comfyui ksampler can run almost any model as long as you have enough ram, ltx2 can run fastly below 4G vram thanks to the reserve vram.

[asomoza]

ComfyUI is a whole app, diffusers is a developer library, you're comparing pears to apples. Diffusers has memory management and optimizations that are very good (some that even ComfyUI doesn't have) but you have to code the app or cli to work with them, they don't just magically work, we don't do any optimizations automatically under the hood for you, you can read the documentation or use some examples from the recipes here

If you want to use a diffusers UI, you can search for them and test them, but you can't compare how ComfyUI manages the resources for you versus vanilla diffusers, they're not the same.

P.S.: LTX2 has a 2-stage inference process that makes the quality better, we still don't have that on diffusers, probably that's why you found the difference in quality.

[zwukong]

Thanks, i know them,cpu offload sequential or layer offload. gradio is widely used in diffuser. LTX2 second stage is just scale up, one stage is enough for testing, while wan2.2 not. In my tests, almost 100 videos, one stage quality is well enough. What i encountered is warped persones , you can even not tell they are human beings😄