GgufLinear: inference-time GGUF matmul on Apple Silicon — llama.cpp parity

docs/source/en/gguf.md

@@ -24,10 +24,10 @@ rendered properly in your Markdown viewer.
 
 The GGUF format also supports many quantized data types (refer to [quantization type table](https://hf.co/docs/hub/en/gguf#quantization-types) for a complete list of supported quantization types) which saves a significant amount of memory, making inference with large models like Whisper and Llama feasible on local and edge devices.
 
-Transformers supports loading models stored in the GGUF format for further training or finetuning. The GGUF checkpoint is **dequantized to fp32** where the full model weights are available and compatible with PyTorch.
+Transformers supports loading models stored in the GGUF format for further training or finetuning. The GGUF checkpoint is **dequantized lazily on the target device** by the conversion-mapping pipeline (see [Adding GGUF support for a new architecture](#adding-gguf-support-for-a-new-architecture) below for how that's plugged in).
 
 > [!TIP]
-> Models that support GGUF include Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, and [more](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/ggml.py)
+> Architectures wired up for GGUF loading include Llama, Mistral, Phi3, Cohere, Qwen2, Qwen3, Deci, StableLM, Starcoder2, Nemotron, Gemma2, Gemma3 (text + multimodal), Bloom, GPT2, Mamba, LFM2, Falcon, Qwen2-MoE, Qwen3-MoE, MiniMax-M2, GPT-OSS, T5, UMT5. The full list can be fetched with `_GGUF_ARCH_CONVERTERS.keys()` in [`modeling_gguf_pytorch_utils.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_gguf_pytorch_utils.py).
 
 Add the `gguf_file` parameter to [`~PreTrainedModel.from_pretrained`] to specify the GGUF file to load.
 
@@ -52,3 +52,82 @@ model.save_pretrained("directory")
 
 !python ${path_to_llama_cpp}/convert-hf-to-gguf.py ${directory}
 ```
+
+## Adding GGUF support for a new architecture
+
+GGUF loading plugs into the same weight-conversion framework that the rest of `transformers` uses for safetensors / Fp8 / etc. To support a new architecture you only need to:
+
+1. **Write a list of `WeightRenaming` / `WeightConverter` rules** in [`src/transformers/modeling_gguf_pytorch_utils.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_gguf_pytorch_utils.py) and register it in `_GGUF_ARCH_CONVERTERS`. You can also just combine existing rules whenever relevant. You then map the `model_type` to the rules.
+2. **Add the `model_type` to `EXPECTED_MODEL_TYPES`** in [`tests/quantization/ggml/test_gguf_arch_coverage.py`](https://github.com/huggingface/transformers/blob/main/tests/quantization/ggml/test_gguf_arch_coverage.py) so the fast regression test enforces it.
+
+### Rule types
+
+For more details you should check the doc [dynamic weight loading](https://huggingface.co/docs/transformers/weightconverter).
+As a reminder it looks like this:
+
+```python
+from transformers.core_model_loading import WeightConverter, WeightRenaming
+
+# Pure key rename (cheap — no op chain). Substring `re.sub` style: escape dots,
+# use alternation when the same target template covers several source names.
+WeightRenaming(r"^blk\.", "model.layers.")
+WeightRenaming(r"\.ffn_(gate|up|down)\.weight", r".mlp.\1_proj.weight")
+
+# With an actual tensor transform (transpose, permute, concat, etc.).
+WeightConverter(
+    source_patterns=r"\.attn_q\.weight",
+    target_patterns=".self_attn.q_proj.weight",
+    operations=[ReversePermuteAttnQ()],
+)
+```
+
+`WeightRenaming`s are applied **sequentially** (each one operates on the previous rule's output), so structural prefix renames (`^blk\.` → `model.layers.`, etc.) should come first and per-tensor renames after. `WeightConverter`s are evaluated after all the renames; the first one whose source pattern matches is selected.
+
+`GGUFDequantize` is automatically prepended to every `WeightConverter`'s op chain by `GGUFQuantizer.update_weight_conversions`, updating the keys to dequantize.
+
+### Existing transform ops
+
+We added a bunch of weight conversion operations that are for now sufficient to cover all the cases. They are in [`src/transformers/gguf_conversion_ops.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/gguf_conversion_ops.py).
+
+| Op | Used for |
+|----|----------|
+| `ReversePermuteAttnQ`, `ReversePermuteAttnK` | Llama-family rope-permuted Q/K weights |
+| `SubtractOne` | Gemma / Nemotron norm weights (stored as `weight + 1` in GGUF) |
+| `Unsqueeze(dim)` | Mamba conv1d, LFM2 shortconv — add a singleton dim |
+| `LogNegate` | Mamba SSM-A: `log(-x)` on load |
+| `BloomReshapeQKVWeight`, `BloomReshapeQKVBias` | Bloom interleaved QKV |
+| `Concatenate(dim)` (from `core_model_loading`) | MoE gate+up merge into `gate_up_proj` |
+| `Transpose` (from `core_model_loading`) | GPT-2 transposed `c_attn` / `c_proj` / `c_fc` |
+
+If your architecture needs a new tensor transform, add it to `gguf_conversion_ops.py` as a `ConversionOps` subclass with `convert(input_dict, source_patterns, target_patterns, **kwargs)` and a `reverse_op` property.
+
+### Example: adding a new Llama-like architecture
+
+For a model whose HF `model_type` is `"my_llama"` and which uses the standard rope-permuted Q/K, plain Llama norms, and `mlp.gate_proj` / `up_proj` / `down_proj`, the rule list is already provided as `_LLAMA_CONVERTERS`. You only need:
+
+```python
+# src/transformers/modeling_gguf_pytorch_utils.py
+_GGUF_ARCH_CONVERTERS = {
+    ...
+    "my_llama": _LLAMA_CONVERTERS,
+}
+```
+
+For a variant with `q_norm` / `k_norm` and attn biases (StableLM style) you compose:
+
+```python
+_MY_LLAMA_CONVERTERS = _LLAMA_CONVERTERS + [
+    WeightRenaming(r"\.attn_(q|k)_norm\.weight", r".self_attn.\1_norm.weight"),
+    WeightRenaming(r"\.attn_(q|k|v)\.bias", r".self_attn.\1_proj.bias"),
+]
+```
+
+For a wholly new layout, start from one of the existing entries (`_BLOOM_CONVERTERS`, `_GPT2_CONVERTERS`, `_T5_CONVERTERS`, …) and adapt the renames + add any required `WeightConverter` ops.
+
+### Dequantization performance
+
+The actual byte-level dequant happens inside the `GGUFDequantize` op via [`integrations/gguf_dequant.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/gguf_dequant.py), a pure-PyTorch port of [city96/ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF) (same kernels diffusers uses). 
+
+It supports Q4_0/Q4_1, Q5_0/Q5_1, Q8_0, Q2_K…Q6_K, IQ4_NL, IQ4_XS, BF16, F16, F32 and runs on CPU or GPU/MPS.
+
+If you add a new quant type, add the corresponding `_dq_<TYPE>` kernel to `gguf_dequant.py` and register it in `_build_dispatch()`.

docs/source/en/main_classes/quantization.md

@@ -117,3 +117,7 @@ Learn how to quantize models in the [Quantization](../quantization) guide.
 ## SinqConfig
 
 [[autodoc]] SinqConfig
+
+## GgufQuantizeConfig
+
+[[autodoc]] GgufQuantizeConfig

src/transformers/__init__.py

@@ -259,6 +259,7 @@
         "FineGrainedFP8Config",
         "FourOverSixConfig",
         "FPQuantConfig",
+        "GgufQuantizeConfig",
         "GPTQConfig",
         "HiggsConfig",
         "HqqConfig",
@@ -775,6 +776,7 @@
     from .utils.quantization_config import FineGrainedFP8Config as FineGrainedFP8Config
     from .utils.quantization_config import FourOverSixConfig as FourOverSixConfig
     from .utils.quantization_config import FPQuantConfig as FPQuantConfig
+    from .utils.quantization_config import GgufQuantizeConfig as GgufQuantizeConfig
     from .utils.quantization_config import GPTQConfig as GPTQConfig
     from .utils.quantization_config import HiggsConfig as HiggsConfig
     from .utils.quantization_config import HqqConfig as HqqConfig

src/transformers/cli/serving/model_manager.py

@@ -228,10 +228,22 @@ def _load_processor(self, model_id_and_revision: str) -> "ProcessorMixin | PreTr
         Args:
             model_id_and_revision: Model ID in ``'model_id@revision'`` format.
         """
+        import os as _os
+
         from transformers import AutoProcessor
+        from transformers.modeling_utils import _maybe_auto_detect_gguf
 
         model_id, revision = model_id_and_revision.split("@", 1)
-        return AutoProcessor.from_pretrained(model_id, revision=revision, trust_remote_code=self.trust_remote_code)
+        kwargs = {"revision": revision, "trust_remote_code": self.trust_remote_code}
+        # GGUF: the tokenizer lives inside the .gguf; point the loader at its dir + filename.
+        if model_id.lower().endswith(".gguf") and _os.path.isfile(model_id):
+            kwargs["gguf_file"] = _os.path.basename(model_id)
+            model_id = _os.path.dirname(model_id)
+        else:
+            _gguf = _maybe_auto_detect_gguf(model_id, revision=revision)
+            if _gguf is not None:
+                kwargs["gguf_file"] = _gguf
+        return AutoProcessor.from_pretrained(model_id, **kwargs)
 
     def _load_model(
         self, model_id_and_revision: str, tqdm_class: type | None = None, progress_callback: Callable | None = None
@@ -260,6 +272,20 @@ def _load_model(
             "tqdm_class": tqdm_class,
         }
 
+        # GGUF: serve receives only a model id. Resolve a single .gguf + pass gguf_file so
+        # AutoConfig + from_pretrained read config/weights from it. Loaders expect a repo/dir
+        # + filename, so for a local .gguf path point model_id at its directory.
+        import os as _os
+        from transformers.modeling_utils import _maybe_auto_detect_gguf
+
+        if model_id.lower().endswith(".gguf") and _os.path.isfile(model_id):
+            model_kwargs["gguf_file"] = _os.path.basename(model_id)
+            model_id = _os.path.dirname(model_id)
+        else:
+            _gguf = _maybe_auto_detect_gguf(model_id, revision=revision)
+            if _gguf is not None:
+                model_kwargs["gguf_file"] = _gguf
+
         if progress_callback is not None:
             progress_callback({"status": "loading", "model": model_id_and_revision, "stage": "config"})
         config = AutoConfig.from_pretrained(model_id, **model_kwargs)
@@ -271,6 +297,11 @@ def _load_model(
 
             return AutoModelForMultimodalLM.from_pretrained(model_id, **model_kwargs)
 
+        if not config.architectures:  # GGUF configs often omit architectures; route by model_type
+            from transformers import AutoModelForCausalLM
+
+            return AutoModelForCausalLM.from_pretrained(model_id, **model_kwargs)
+
         architecture = getattr(transformers, config.architectures[0])
         return architecture.from_pretrained(model_id, **model_kwargs)