Repository navigation
[docs] gguf #46357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[docs] gguf #46357
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,28 +25,47 @@ rendered properly in your Markdown viewer. | |
|
|
||
| ## Load GGUF models | ||
|
|
||
| Install [kernels](https://huggingface.co/docs/kernels/index), otherwise the packed path falls back to full [dequantization](#dequantize) at load. | ||
|
|
||
| ```bash | ||
| pip install kernels | ||
| ``` | ||
|
|
||
| Weights stay packed when the Hub kernel [transformers-community/ggml-quantization](https://huggingface.co/transformers-community/ggml-quantization) is available. The loader defaults to MPS when that kernel is present and runs matmuls directly on the packed blocks. If the kernel isn't available, the model is dequantized at load. | ||
|
|
||
| ```py | ||
| import torch | ||
|
|
||
| from transformers import AutoModelForCausalLM, AutoTokenizer | ||
|
|
||
| model_id = "unsloth/Qwen3.5-4B-GGUF" | ||
| filename = "Qwen3.5-4B-Q4_K_M.gguf" | ||
|
|
||
| model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename) | ||
| model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename, dtype=torch.float32) | ||
| tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename) | ||
| ``` | ||
|
|
||
| The weights only stay in their GGUF blocks on Metal (MPS) devices, where the [llama.cpp](https://github.com/ggerganov/llama.cpp) kernels, fetched from the Hub, run the matmuls directly on the packed blocks to keep inference fast. | ||
| The same load works for Qwen3.5 MoE. | ||
|
|
||
| ```py | ||
| moe_model_id = "unsloth/Qwen3.5-35B-A3B-GGUF" | ||
| moe_filename = "Qwen3.5-35B-A3B-Q4_K_M.gguf" | ||
|
|
||
| moe_model = AutoModelForCausalLM.from_pretrained( | ||
| moe_model_id, gguf_file=moe_filename, dtype=torch.float32 | ||
| ) | ||
| moe_tokenizer = AutoTokenizer.from_pretrained(moe_model_id, gguf_file=moe_filename) | ||
| ``` | ||
|
Comment on lines
+46
to
+54
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. not yet, but i've opened a bit that enables it ! so we should merge this pr after the other PRs gets merged
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. good to merge now that #48529 is merged or are there more/other PRs we should also wait for? |
||
|
|
||
| Right now, the only architecture supported is Qwen3.5. Everything else falls back. On another device or quantization type, the model is [dequantized](#dequantize) at load time, and an architecture that isn't supported yet goes through the legacy loader. | ||
| The packed path currently supports Qwen3.5 and Qwen3.5 MoE. Use `dtype=torch.float32` for packed loads. The loader warns when another dtype is requested because the kernels compute in float32. Other architectures go through the legacy loader. | ||
|
stevhliu marked this conversation as resolved.
Resolved
|
||
|
|
||
| ## Attention | ||
|
|
||
| On Metal, attention has a ggml kernel too: [ggml-attn](https://huggingface.co/transformers-community/ggml-attn), the same | ||
| flash attention llama.cpp runs, for both decode and prefill. | ||
| On Metal, with kernels installed, attention can use [ggml-attn](https://huggingface.co/transformers-community/ggml-attn), the same flash-attention kernel llama.cpp uses for decode and prefill. | ||
|
|
||
| ```py | ||
| model = AutoModelForCausalLM.from_pretrained( | ||
| model_id, gguf_file=filename, attn_implementation="transformers-community/ggml-attn" | ||
| model_id, gguf_file=filename, dtype=torch.float32, attn_implementation="transformers-community/ggml-attn" | ||
|
stevhliu marked this conversation as resolved.
Resolved
|
||
| ) | ||
| ``` | ||
|
|
||
|
|
@@ -65,16 +84,16 @@ model = AutoModelForCausalLM.from_pretrained( | |
| ) | ||
| ``` | ||
|
|
||
| The model that comes out is a regular dense model, so this is also how you take a GGUF checkpoint into the dtype you passed. | ||
| You get a regular dense model in the `dtype` you passed. | ||
|
|
||
| Architectures other than Qwen3.5 are read by the legacy loader which dequantize the model also. | ||
| Architectures other than Qwen3.5 and Qwen3.5 MoE go through the legacy loader, which always dequantizes. | ||
|
|
||
| > [!TIP] | ||
| > The legacy loader supports Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, and [more](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/ggml.py). | ||
|
|
||
| ## Serve | ||
|
|
||
| [`transformers serve`](../serve-cli/serving) names a GGUF model `<repo>:<file>.gguf`, since a repository holds | ||
| [transformers serve](../serve-cli/serving) lists each `.gguf` file in a repository as its own model. A GGUF model is named `<repo>:<file>.gguf`, since a repository holds | ||
| several quantizations and the id has to say which one to load. Requests name it the same way. | ||
|
|
||
| ```shell | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.