Skip to content
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
9789b12
Add an FP8 backend to `QuantizedCache`
jiqing-feng Sep 10, 2026
64cfc5d
Reorder the FP8 states for beam search
jiqing-feng Sep 10, 2026
87ab14a
Use an early return in `Fp8QuantizedLayer.reorder_cache`
jiqing-feng Sep 10, 2026
a7cc8e3
Measure the quantized cache backends accurately
jiqing-feng Sep 14, 2026
8882b42
Merge branch 'main' into fp8-kv-cache
jiqing-feng Sep 17, 2026
2b7a9eb
Merge branch 'main' into fp8-kv-cache
jiqing-feng Oct 6, 2026
bc126c1
Reset the FP8 scales, and cover the FP8 backend in the quantized laye…
jiqing-feng Oct 6, 2026
5eaeb32
Merge branch 'main' into fp8-kv-cache
jiqing-feng Oct 6, 2026
4fc4f00
Merge branch 'main' into fp8-kv-cache
jiqing-feng Oct 6, 2026
4743c39
Address review on the FP8 quantized cache layer
jiqing-feng Oct 16, 2026
e9e889d
Trim the comments down to one line
jiqing-feng Oct 16, 2026
05ef372
Store the FP8 scale alongside the states it quantizes
jiqing-feng Oct 16, 2026
bc31eda
Say what the `.contiguous()` in `lazy_initialization` is actually for
jiqing-feng Oct 16, 2026
ed19267
Address @vasqu's review
jiqing-feng Oct 19, 2026
7018cc4
One-line the docstrings and comments that fit on one line
jiqing-feng Oct 19, 2026
862a643
Merge branch 'main' into fp8-kv-cache
jiqing-feng Oct 19, 2026
b4e172b
Keep the FP8 state initialization out of the shared quantized layer
jiqing-feng Oct 19, 2026
02dc2e8
Split the quantized layer into a base and a group-wise class
jiqing-feng Oct 20, 2026
b138c83
Note which subtree a future fp8 variation belongs to
jiqing-feng Oct 20, 2026
780cc9d
Keep the group-wise signature on QuantizedLayer and name the new base
jiqing-feng Oct 21, 2026
08c5bc7
Declare the quantized states in the base layer, and group the quantiz…
jiqing-feng Oct 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/source/en/internal/generation_utils.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,10 @@ A [`StoppingCriteria`] can be used to change when to stop generation (other than
- update
- lazy_initialization

[[autodoc]] Fp8QuantizedLayer
- update
- lazy_initialization

[[autodoc]] Cache
- update
- early_initialization
Expand Down
18 changes: 17 additions & 1 deletion docs/source/en/kv_cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,10 +176,11 @@ responses = tokenizer.batch_decode(out[:,-28:], skip_special_tokens=True)

## Quantized cache

The [`QuantizedCache`] reduces memory requirements by quantizing the KV values to a lower precision. [`QuantizedCache`] currently supports two quantization backends:
The [`QuantizedCache`] reduces memory requirements by quantizing the KV values to a lower precision. [`QuantizedCache`] currently supports three quantization backends:

- `hqq` supports int1, int2, int3, int4, and int8 datatypes.
- `quanto` supports int2 and int4 datatypes. This is the default quantization backend.
- `fp8` stores the KV values in 8-bit floating point (`float8_e4m3fn`). It requires no extra dependency, and quantizes per-tensor rather than per-group, so it rejects `nbits`, `axis_key`, `axis_value`, `q_group_size` and `residual_length`.

> [!WARNING]
> Quantizing the cache can harm latency if the context length is short and there is enough GPU memory available for generation without enabling cache quantization. Try to find a balance between memory efficiency and latency.
Expand Down Expand Up @@ -238,6 +239,21 @@ print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
I like rock music because it's loud and energetic. It's a great way to express myself and rel
```

The `fp8` backend does not take any additional parameter, and raises if it is given one of the group-wise ones.

```py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", dtype=torch.float16, device_map="auto")
inputs = tokenizer("I like rock music because", return_tensors="pt").to(model.device)

out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation="quantized", cache_config={"backend": "fp8"})
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
I like rock music because it is powerful and emotional. I also like classical music because it is elegant and sophisticated. I enjoy
```

## Encoder-decoder cache

[`EncoderDecoderCache`] is designed for encoder-decoder models. It manages both the self-attention and cross-attention caches to ensure storage and retrieval of previous kv pairs. It is possible to individually set a different cache type for the encoder and decoder.
Expand Down
2 changes: 2 additions & 0 deletions src/transformers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,7 @@
"DynamicIndexedLayer",
"DynamicLayer",
"EncoderDecoderCache",
"Fp8QuantizedLayer",
"HQQQuantizedLayer",
"QuantizedCache",
"QuantoQuantizedLayer",
Expand Down Expand Up @@ -468,6 +469,7 @@
from .cache_utils import DynamicIndexedLayer as DynamicIndexedLayer
from .cache_utils import DynamicLayer as DynamicLayer
from .cache_utils import EncoderDecoderCache as EncoderDecoderCache
from .cache_utils import Fp8QuantizedLayer as Fp8QuantizedLayer
from .cache_utils import HQQQuantizedLayer as HQQQuantizedLayer
from .cache_utils import QuantizedCache as QuantizedCache
from .cache_utils import QuantoQuantizedLayer as QuantoQuantizedLayer
Expand Down
166 changes: 140 additions & 26 deletions src/transformers/cache_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -690,6 +690,34 @@ def reorder_cache(self, beam_idx: torch.LongTensor) -> None:


class QuantizedLayer(DynamicLayer):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slightly breaking thos if users relied on QuantizedLayer so let's add 🚨 to the title and describe this

Or do you think we can make a proper BC version without making it too ugly? 😅

@jiqing-feng jiqing-feng Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a trade-off between BC and naming, pick whichever you prefer. 780cc9d has the BC one.

QuantizedLayer keeps the exact signature it has on main, down to _quantize(self, tensor, axis), and the new BaseQuantizedLayer holds what all three backends share. A subclass written against main needs no change.

What it costs is that QuantizedLayer now means "group-wise quantized layer" rather than "any quantized layer", so Fp8QuantizedLayer is not an instance of it. The isinstance check in generation_vibevoice.py has to move to the new base, otherwise FP8 layers stop being skipped and the code indexes into their empty keys. That's the only call site in the repo, and every new backend now has to pick the right one of the two to inherit from.

Outside the repo nothing breaks either way: isinstance(layer, QuantizedLayer) keeps matching quanto and hqq, and couldn't have been handed an FP8 layer before this PR.

If you'd rather keep the clearer naming, I'll revert to b138c83 and flag the break instead.

Let me know your choice (these 2 above or others) so I can rewrite the title and PR statement.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the current option is quite nice, maybe slightly confusing but not by much imo

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the 🚨 then as well no?

"""
Base class for the cache layers that store their key and value states in a quantized form, in
`_quantized_keys` and `_quantized_values`. Their layout is left to the backend, so the sequence length is
tracked separately, in `cumulative_length`.
"""

def __init__(self):
super().__init__()
self.cumulative_length = 0

@abstractmethod
def _quantize(self, tensor): ...

@abstractmethod
def _dequantize(self, q_tensor): ...

def reset(self) -> None:
"""Resets the cache values while preserving the objects."""
super().reset()
# The quantized states are dropped instead of zeroed, so that the next `update` quantizes from scratch
self._quantized_keys = self._quantized_values = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldnt these also not be declared in init?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, they were only ever assigned in reset and lazy_initialization, so a freshly built layer had no such attribute at all. Declared both in BaseQuantizedLayer.__init__ next to cumulative_length, and kept the reset assignment, which is what actually drops the states.


def get_seq_length(self) -> int:
"""Returns the sequence length of the cached states."""
return self.cumulative_length


class GroupWiseQuantizedLayer(QuantizedLayer):
"""
A quantized layer similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://huggingface.co/papers/2402.02750).
It allows the model to generate longer sequence length without allocating too much memory for the key and value caches by
Expand All @@ -702,6 +730,9 @@ class QuantizedLayer(DynamicLayer):

Note that a beam reordering is only applied to the quantized states on the next `update`, so they may be left in a
stale batch order in between, see `reorder_cache`.

What is shared here is the residual window, not the grouping itself: a backend whose quantized states can be
appended to, as FP8 does, has no use for it and subclasses `QuantizedLayer` directly instead.
"""

def __init__(
Expand All @@ -718,10 +749,12 @@ def __init__(
self.axis_value = axis_value
self.q_group_size = q_group_size
self.residual_length = residual_length
self.cumulative_length = 0
# Beam reordering of the quantized states is deferred until the next `update`, see `reorder_cache`
self._pending_beam_idx = None

@abstractmethod
def _quantize(self, tensor, axis): ...

def update(
self, key_states: torch.Tensor, value_states: torch.Tensor, *args, **kwargs
) -> tuple[torch.Tensor, torch.Tensor]:
Expand Down Expand Up @@ -765,12 +798,6 @@ def update(

return keys_to_return, values_to_return

@abstractmethod
def _quantize(self, tensor, axis): ...

@abstractmethod
def _dequantize(self, q_tensor): ...

def reorder_cache(self, beam_idx: torch.LongTensor) -> None:
"""Reorders this layer's cache for beam search."""
if not self.is_initialized:
Expand Down Expand Up @@ -798,16 +825,10 @@ def _apply_pending_reorder(self, keys: torch.Tensor, values: torch.Tensor) -> tu
def reset(self) -> None:
"""Resets the cache values while preserving the objects."""
super().reset()
# The quantized states are dropped instead of zeroed, so that the next `update` quantizes from scratch
self._quantized_keys = self._quantized_values = None
self._pending_beam_idx = None

def get_seq_length(self) -> int:
"""Returns the sequence length of the cached states."""
return self.cumulative_length


class QuantoQuantizedLayer(QuantizedLayer):
class QuantoQuantizedLayer(GroupWiseQuantizedLayer):
def __init__(
self,
nbits: int = 4,
Expand Down Expand Up @@ -862,7 +883,7 @@ def _dequantize(self, qtensor):
return qtensor.dequantize()


class HQQQuantizedLayer(QuantizedLayer):
class HQQQuantizedLayer(GroupWiseQuantizedLayer):
def __init__(
self,
nbits: int = 4,
Expand Down Expand Up @@ -919,6 +940,76 @@ def _dequantize(self, qtensor):
return tensor


class Fp8QuantizedLayer(QuantizedLayer):
"""
A quantized layer storing the key and value states in 8-bit floating point. The per-tensor scale is frozen
after the first `update`, so new states are appended as-is: no residual cache, no re-quantization.

A finer-grained variant belongs here rather than under `GroupWiseQuantizedLayer`, as grouping along the head
dimension keeps states appendable and only turns the stored scale into a tensor.
"""

fp8_dtype = torch.float8_e4m3fn
fp8_max = torch.finfo(fp8_dtype).max
Comment on lines +953 to +954

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to guard for older torch versions e.g. 2.6?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. setup.py on main requires torch>=2.6, and Ilyas tested the fp8 path on 2.6 earlier in this review — it only breaks on 2.5. I also ran the torch.cat check on 2.8 / 2.10 / 2.11 / 2.12 / 2.13 / 2.14, CPU and CUDA.


def lazy_initialization(self, key_states: torch.Tensor, value_states: torch.Tensor) -> None:
super().lazy_initialization(key_states, value_states)
self._quantized_keys = self._quantize(key_states)
self._quantized_values = self._quantize(value_states)

def _quantize(self, tensor, scale=None):
"""Quantize `tensor` to FP8, calibrating the scale on its amplitude if it is not provided."""
if scale is None:
# `clamp` guards against an all-zeros tensor, which would give a null scale
scale = (tensor.abs().amax().float() / self.fp8_max).clamp(min=torch.finfo(torch.float32).tiny)
return (tensor.float() / scale).to(self.fp8_dtype), scale

def _dequantize(self, qtensor):
quant_tensor, scale = qtensor
return (quant_tensor.float() * scale).to(self.dtype)
Comment on lines +968 to +970

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.


def update(
self, key_states: torch.Tensor, value_states: torch.Tensor, *args, **kwargs
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Update the key and value caches in-place, and return the necessary keys and value states.

Args:
key_states (`torch.Tensor`): The new key states to cache.
value_states (`torch.Tensor`): The new value states to cache.

Returns:
tuple[`torch.Tensor`, `torch.Tensor`]: The key and value states.
"""
self.cumulative_length += key_states.shape[-2]

if not self.is_initialized:
self.lazy_initialization(key_states, value_states)
return key_states, value_states
Comment on lines +988 to +989

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it makes sense to override lazy_initialization to init scales as well

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. update is now only the steady-state path.


quantized_keys, key_scale = self._quantized_keys
quantized_values, value_scale = self._quantized_values
new_keys, _ = self._quantize(key_states, scale=key_scale)
new_values, _ = self._quantize(value_states, scale=value_scale)
self._quantized_keys = (torch.cat([quantized_keys, new_keys], dim=-2), key_scale)
self._quantized_values = (torch.cat([quantized_values, new_values], dim=-2), value_scale)

keys_to_return = self._dequantize(self._quantized_keys)
values_to_return = self._dequantize(self._quantized_values)
Comment on lines +998 to +999

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ig at some point we could involve FA with fp8 support right? so no real dequant etc

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and the kernel is already there — kernels-community/flash-attn3 builds the *_e4m3_sm90 instantiations and takes q_descale/k_descale/v_descale. The missing piece is ours: modeling_flash_attention_utils.py has no way to pass a descale, so there's nothing to hand fp8 states to yet. SDPA can't take them either ("mul_cuda" not implemented for 'Float8_e4m3fn'), hence the dequantization.

Two notes for whoever wires it up. FA3 wants the descale as a (batch_size, num_heads_k) tensor, so a per-tensor scale goes in expanded — the kernel is already finer-grained than this layer. And it's Hopper-only; Intel XPU falls back to flash-attn2, which has no fp8 kernels. I'm looking at the XPU side next, so I'd rather the storage layout here didn't end up CUDA-shaped.

return keys_to_return, values_to_return

def reorder_cache(self, beam_idx: torch.LongTensor) -> None:
"""Reorders the cache for beam search. The per-tensor scale lets the FP8 states be reordered as-is."""
if not self.is_initialized:
return

quantized_keys, key_scale = self._quantized_keys
quantized_values, value_scale = self._quantized_values
beam_idx = beam_idx.to(quantized_keys.device)
self._quantized_keys = (quantized_keys.index_select(0, beam_idx), key_scale)
self._quantized_values = (quantized_values.index_select(0, beam_idx), value_scale)


class LinearAttentionCacheLayerMixin(ABC):
"""Base, abstract class for a linear attention single layer's cache."""

Expand Down Expand Up @@ -1935,7 +2026,7 @@ class QuantizedCache(Cache):

Args:
backend (`str`):
The quantization backend to use. One of `("quanto", "hqq").
The quantization backend to use. One of `("quanto", "hqq", "fp8")`.
config (`PreTrainedConfig`):
The config of the model for which this Cache will be used.
nbits (`int`, *optional*, defaults to 4):
Expand All @@ -1947,26 +2038,52 @@ class QuantizedCache(Cache):
q_group_size (`int`, *optional*, defaults to 64):
Quantization is done per-channel according to a set `q_group_size` for both keys and values.
residual_length (`int`, *optional*, defaults to 128):
Maximum capacity for the original precision cache
Maximum capacity for the original precision cache.

The last five arguments configure the group-wise integer backends. The `fp8` backend quantizes per-tensor and
rejects them.
"""

def __init__(
self,
backend: str,
config: PreTrainedConfig,
nbits: int = 4,
axis_key: int = 0,
axis_value: int = 0,
q_group_size: int = 64,
residual_length: int = 128,
nbits: int | None = None,
axis_key: int | None = None,
axis_value: int | None = None,
q_group_size: int | None = None,
residual_length: int | None = None,
):
group_wise_kwargs = {
"nbits": nbits,
"axis_key": axis_key,
"axis_value": axis_value,
"q_group_size": q_group_size,
"residual_length": residual_length,
}
if backend == "quanto":
layer_class = QuantoQuantizedLayer
elif backend == "hqq":
layer_class = HQQQuantizedLayer
elif backend == "fp8":
layer_class = Fp8QuantizedLayer
else:
raise ValueError(f"Unknown quantization backend `{backend}`")

# A backend either takes all of the group-wise options or none of them. A future group-wise fp8 variant
# would need this to be per-backend instead.
if backend == "fp8":
unsupported = [name for name, value in group_wise_kwargs.items() if value is not None]
if len(unsupported) > 0:
raise ValueError(
f"The `fp8` backend quantizes per-tensor and does not support {unsupported}. Use the `quanto` or "
"`hqq` backend to quantize group-wise."
)
layer_kwargs = {}
else:
# The unset ones are left out, so that the layer's own defaults apply
layer_kwargs = {name: value for name, value in group_wise_kwargs.items() if value is not None}

config = config.get_text_config(decoder=True)
layer_types, _ = get_layer_types_and_kwargs(config)
invalid_layer_types = set(layer_types) - {"full_attention"}
Expand All @@ -1975,10 +2092,7 @@ def __init__(
"`QuantizedCache` is only supported for models with only full attention layers. We found the following invalid layer "
f"types: {invalid_layer_types}"
)
layers = [
layer_class(nbits, axis_key, axis_value, q_group_size, residual_length)
for _ in range(config.num_hidden_layers)
]
layers = [layer_class(**layer_kwargs) for _ in range(config.num_hidden_layers)]
super().__init__(layers=layers)


Expand Down
Loading
Loading