From 8b618dc913b3a86e1f7ac9d8e8a9a06df2588088 Mon Sep 17 00:00:00 2001 From: MerkyorLynn Date: Mon, 8 Jun 2026 00:54:04 +0800 Subject: [PATCH] docs: add NVFP4 conversion and benchmarking notes Adds docs/nvfp4.md describing the NVFP4 weight format, how to obtain an NVFP4 GGUF (converting ModelOpt / compressed-tensors checkpoints via convert_hf_to_gguf.py), when the native FP4 path is selected (blackwell_mma_available()), and how to benchmark it against Q4_K_M. Also adds a one-line pointer from tools/quantize/README.md. No code changes. --- docs/nvfp4.md | 99 ++++++++++++++++++++++++++++++++++++++++ tools/quantize/README.md | 1 + 2 files changed, 100 insertions(+) create mode 100644 docs/nvfp4.md diff --git a/docs/nvfp4.md b/docs/nvfp4.md new file mode 100644 index 000000000000..1b49e4531372 --- /dev/null +++ b/docs/nvfp4.md @@ -0,0 +1,99 @@ +# NVFP4 quantization + +NVFP4 is NVIDIA's 4-bit floating-point weight format. In llama.cpp it is +represented by `GGML_TYPE_NVFP4`, with packed `E2M1` values and per-block +`E4M3`/UE4M3 scale data. + +llama.cpp support is split into two parts: + +- GGUF / ggml type support was added in + [#19769](https://github.com/ggml-org/llama.cpp/pull/19769). +- A native CUDA path for Blackwell FP4 matrix-matrix workloads was added in + [#22196](https://github.com/ggml-org/llama.cpp/pull/22196). + +This page summarizes when NVFP4 is useful and how to compare it with mature +4-bit formats such as `Q4_K_M`. + +## When to try NVFP4 + +NVFP4 is most useful when at least one of the following is true: + +- You are converting or running a Hugging Face checkpoint that is already + distributed in ModelOpt or compressed-tensors NVFP4 format. +- You are running on a CUDA Blackwell build and care about prompt processing or + batched matrix-matrix workloads where the native FP4 path can be selected. +- You want to benchmark NVIDIA's FP4 checkpoint ecosystem against llama.cpp's + existing integer 4-bit formats. + +NVFP4 is not automatically faster for every workload. One-token-at-a-time text +generation is often memory-bandwidth or vector-dot limited, so it may not see +the same gains as prompt processing or batched workloads. Always benchmark +prompt processing and text generation separately. + +## Hardware and backend notes + +- Native CUDA FP4 acceleration requires a Blackwell CUDA build. The CUDA backend + selects the native FP4 path for MXFP4 / NVFP4 matrix-matrix workloads when + `blackwell_mma_available()` is true for the active build and device. +- Other backends can still load NVFP4 GGUF tensors through their supported + dequantization / vector-dot paths, but performance depends on the backend and + workload. +- Support is still evolving. If a device is Blackwell-class but performance is + unexpected, check the build architecture flags, backend log output, and + `llama-bench` results before assuming the native FP4 path is active. + +## NVFP4 versus Q4_K_M + +`Q4_K_M` remains a strong default for general interactive use because it has +mature kernels and broad backend coverage. + +NVFP4 is a good candidate when you want to preserve an upstream NVFP4 checkpoint +format or benchmark Blackwell FP4 hardware. On single-stream interactive decode, +compare against `Q4_K_M` rather than assuming NVFP4 will win. On Blackwell +prompt processing or batched workloads, NVFP4 may be more attractive because +the native FP4 path can reduce compute cost. + +Quality also depends on the model, calibration recipe, tensor selection, and +whether the source checkpoint was trained or calibrated for NVFP4. There is no +universal quality ordering between NVFP4 and `Q4_K_M`; run task-specific +evaluation for your model. + +## Conversion and usage + +For ModelOpt or compressed-tensors NVFP4 checkpoints on Hugging Face, use +`convert_hf_to_gguf.py` to convert the checkpoint to GGUF. The converter detects +NVFP4 metadata and repacks supported NVFP4 tensors into the GGUF layout used by +llama.cpp. + +Example: + +```bash +python3 convert_hf_to_gguf.py \ + --outfile model-nvfp4.gguf \ + --remote / +``` + +Then run the resulting GGUF as usual: + +```bash +./build/bin/llama-cli -m model-nvfp4.gguf -p "Hello" +``` + +If you are quantizing a high-precision GGUF locally, check +`llama-quantize --help` in your build for the currently supported quantization +targets. Not every build exposes NVFP4 as a direct `llama-quantize` target. + +## Benchmarking checklist + +Use `llama-bench` or your normal serving benchmark to record both prompt +processing and text generation: + +```bash +./build/bin/llama-bench -m model-nvfp4.gguf -p 512 -n 128 +./build/bin/llama-bench -m model-q4_k_m.gguf -p 512 -n 128 +``` + +When comparing formats, keep the same model architecture, prompt length, +generation length, batch/concurrency setting, backend, and GPU build flags. +Report prompt processing and text generation numbers separately because NVFP4 +can affect them differently. diff --git a/tools/quantize/README.md b/tools/quantize/README.md index 27384bebf697..fe473e51df80 100644 --- a/tools/quantize/README.md +++ b/tools/quantize/README.md @@ -39,6 +39,7 @@ Notes: - In the usual case where the model is distributed in 16-bit format, `--outtype auto` (or omitting `--outtype` entirely) also works well. - If you have previously downloaded the model locally, specify the directory and remove the `--remote` flag. - For compatibility reasons, the Python requirements install transformers 4, but more and more models (like Gemma 4) require transformers 5. You can safely `pip install -U transformers` to get the latest version. +- For NVIDIA NVFP4 (FP4) checkpoints distributed in ModelOpt or compressed-tensors format, see [NVFP4 quantization](../../docs/nvfp4.md) for conversion and benchmarking notes. ## Quantize the GGUF