[bojanstef]

Fixes #46424 x

What

GenerationConfig.cache_config is documented as carrying "arguments used in the key-value cache class", but on the static-cache path it is silently ignored. Setting cache_config={"max_cache_len": N} together with cache_implementation="static" produces a cache sized to max_length - 1, not N:

gen = GenerationConfig(max_new_tokens=16, cache_implementation="static",
                       cache_config={"max_cache_len": 2048}, do_sample=False)
model.generate(input_ids, generation_config=gen)   # prompt_len = 32
model._cache.max_cache_len      # 47  (== max_length-1), not the requested 2048

The practical cost (from the issue): a second generate() with a larger max_new_tokens pays a full StaticCache realloc + torch.compile recompile, because the auto-allocated cache was sized for the first call. The documented way to avoid that — pin the size via cache_config — is exactly the path that doesn't work today.

Root cause

In GenerationMixin._prepare_cache_for_generation (src/transformers/generation/utils.py), cache_config is read only inside the elif generation_config.cache_implementation == "quantized" branch. The static branch sizes the cache straight from the max_cache_length argument (= generation_config.max_length - 1, computed by the caller) and never consults cache_config:

if generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS:
    ...
    model_kwargs[cache_name] = self._prepare_static_cache(
        ...
        max_cache_len=max_cache_length,   # <-- cache_config never read here
    )
elif generation_config.cache_implementation == "quantized":
    ...
    cache_config = generation_config.cache_config if ... else {}   # <-- only reader

A runtime trace of the repro confirms it: max_cache_length arrives as 47, the static branch is taken, and neither the quantized-branch condition nor the cache_config read ever executes (trace below).

When it was introduced (not a regression)

git bisect has no "good" commit here — the static path has never honored cache_config. cache_config was introduced as a QuantizedCache config object, and every refactor since has kept it quantized-only:

$ git log -S cache_config -- src/transformers/generation/utils.py   # earliest
d583f1317b  2024-05-23  Quantized KV Cache (#30483)              # cache_config is born, quantized-only
1687954f5a  2026-02-04  🚨 Generation cache preparation (#43679) # current branch structure

At both #43679 and its parent commit, cache_config is read only on the quantized branch. So this is a long-standing gap dating to the field's origin (#30483), not a regression.

Fix

Thread cache_config["max_cache_len"] into the static path, guarded with max() so the cache is never under-sized for the current call:

if generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS:
    ...
    static_max_cache_len = max_cache_length
    requested_max_cache_len = (generation_config.cache_config or {}).get("max_cache_len")
    if requested_max_cache_len is not None:
        static_max_cache_len = max(max_cache_length, requested_max_cache_len)
    model_kwargs[cache_name] = self._prepare_static_cache(
        ...
        max_cache_len=static_max_cache_len,
    )

When cache_config is None or carries no max_cache_len, the value is unchanged — no behavior change for the default path. max(...) means an explicit-but-too-small max_cache_len can never shrink the cache below what the current call needs.

Tests

Added tests/utils/test_cache_utils.py::CacheTest::test_static_cache_honors_cache_config_max_cache_len (in the existing no-download / no-GPU CacheTest class): builds a tiny random Llama, runs generate() with cache_implementation="static" and cache_config={"max_cache_len": 2048}, and asserts the allocated cache is sized to 2048.

• With this fix: passes.
• Without it: fails with AssertionError: 47 != 2048 (the bug).

Runtime trace — before the fix (the diagnosis)

Captured by instrumenting src/transformers/generation/utils.py and running the repro. file:line are pristine source coordinates.

#10  utils.py:2185  → generate(…)  generation_config = { "cache_config": {"max_cache_len": 2048}, "cache_implementation": "static", … }
#190 utils.py:2549  max_cache_length = 47                                                       # = max_length - 1
#192 utils.py:1861  → _prepare_cache_for_generation(… max_cache_length=47)
#207 utils.py:1922  generation_config.cache_implementation == "offloaded" = False
#208 utils.py:1925  generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS = True
#209 utils.py:1926  generation_config.cache_implementation in DEPRECATED_STATIC_CACHE_IMPLEMENTATIONS = False
#221 utils.py:1955  self.config.is_encoder_decoder = False

The complete set of events between lines 1920–1960 is the four shown above. There is no event at line 1938 (elif … == "quantized") and no event at line 1945 (cache_config = …): the only code that reads cache_config is never reached on the static path, so the requested 2048 is structurally unreachable and 47 sizes the cache.

Runtime trace — after the fix (verification)

#208 utils.py:1925  generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS = True
#210 utils.py:1936  static_max_cache_len    = 47       # starts at max_length - 1
#211 utils.py:1937  requested_max_cache_len = 2048     # static path now reads cache_config
#212 utils.py:1938  requested_max_cache_len is not None = True
#213 utils.py:1939  static_max_cache_len    = 2048     # max(47, 2048) → cache sized 2048

Who can help

@ArthurZucker @Cyrilvallez

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46700&sha=b2fb78

[github-actions]

CI Dashboard: View test results in Grafana

[bojanstef]

Ready for review. The single failing check (tests_processors on CircleCI) is an unrelated network flake, not a regression from this change:

• Test: tests/models/qwen3_vl/test_processing_qwen3_vl.py::Qwen3VLProcessorTest::test_apply_chat_template_video_frame_sampling
• Error: httpx.ConnectError: [SSL: UNEXPECTED_EOF_WHILE_READING] while fetching a remote video asset

This diff only touches the static-cache path in generation/utils.py and adds a no-download test in tests/utils/test_cache_utils.py, with no connection to Qwen3-VL processing or any network I/O. All GitHub Actions checks and the other CircleCI jobs are green. Happy to rebase or re-push if you'd like a fresh CI run.

[Rocketknight1]

Immediate anti-spam block, sorry! There was a PR open at #46446 for the same issue which your bot didn't bother to check, then it pinged two core maintainers to ensure it wasted the maximum amount of everyone's time.

[bojanstef-rs]

@Rocketknight1 ah, I thought I checked but I clearly missed #46446

In my defense I took care to reproduce the bug and include runtime traces

Thanks anways