Repository navigation
[docs] gguf - #46357
[docs] gguf#46357
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
LGTM ! just a small nit as i prefer to not talk about the float32 in the docs. When loading a gguf model, I think the user expect from transformers that everything is loaded correctly and the user don't need to deal with this ! Of course, it is different when the user dequantize the model.
| The same load works for Qwen3.5 MoE. | ||
|
|
||
| ```py | ||
| moe_model_id = "unsloth/Qwen3.5-35B-A3B-GGUF" | ||
| moe_filename = "Qwen3.5-35B-A3B-Q4_K_M.gguf" | ||
|
|
||
| moe_model = AutoModelForCausalLM.from_pretrained( | ||
| moe_model_id, gguf_file=moe_filename, dtype=torch.float32 | ||
| ) | ||
| moe_tokenizer = AutoTokenizer.from_pretrained(moe_model_id, gguf_file=moe_filename) | ||
| ``` |
There was a problem hiding this comment.
not yet, but i've opened a bit that enables it ! so we should merge this pr after the other PRs gets merged
There was a problem hiding this comment.
good to merge now that #48529 is merged or are there more/other PRs we should also wait for?
CI recapDashboard: View test results in Grafana |
* update * moe * review
Updates GGUF docs with mention of kernels (follow-up to #47779) and MoE support (#48529)