Skip to content

Fix compressed-tensors loading of KV-cache-only checkpoints - #50030

Merged
SunMarc merged 1 commit into
huggingface:mainfrom
kylesayrs:fix-compressed-tensors-kv-cache-only-loading
Oct 27, 2026
Merged

Fix compressed-tensors loading of KV-cache-only checkpoints#50030
SunMarc merged 1 commit into
huggingface:mainfrom
kylesayrs:fix-compressed-tensors-kv-cache-only-loading

Conversation

@kylesayrs

@kylesayrs kylesayrs commented Oct 27, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

🤗 Transformers CI

Background

#49548 introduced a bug where elements from the CT config_groups are checked unconditionally, leading to a bug when loading KV-cache-quantization-only models

tests/llmcompressor/transformers/kv_cache/test_kv_cache.py:131:
>       model = AutoModelForCausalLM.from_pretrained(output_dir, device_map="meta")
../TEST_VENV/lib/python3.13/site-packages/transformers/quantizers/quantizer_compressed_tensors.py:175: in serialized_names
    """Resolve which config group quantizes the MoE experts. Mixed configs quantize
>       return groups[0]
E       IndexError: list index out of range
../TEST_VENV/lib/python3.13/site-packages/transformers/integrations/compressed_tensors.py:42: IndexError

Purpose

  • Fix CT loading of KV-cache-quantization-only models

Changes

  • Explicitly check if the model does not have config groups before checking if it has an fp8 scheme

Testing

  • Added e2e KV-cache-quantization-only test

@kylesayrs
Fix compressed-tensors loading of KV-cache-only checkpoints
1a02371 
A compressed-tensors checkpoint that only quantizes the KV cache
(`kv_cache_scheme` set, no `config_groups`) crashed at load time:
`get_experts_scheme` indexed `groups[0]` on an empty list from
`serialized_names` and `get_load_operations`. Guard both on empty
config groups, like the existing sparsity-only case, and add a CI
test on inference-optimization/TinyQwen3-KV-Cache-Only that checks
the scales load onto every attention module.
@github-actions

github-actions Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: compressed_tensors_integration

@kylesayrs
kylesayrs marked this pull request as ready for review October 27, 2026 15:33
@github-actions
github-actions Bot requested review from SunMarc and ydshieh October 27, 2026 15:34
@kylesayrs kylesayrs mentioned this pull request Oct 27, 2026
Merged
SunMarc
SunMarc approved these changes Oct 27, 2026
View reviewed changes

@SunMarc SunMarc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update !

@SunMarc
SunMarc enabled auto-merge October 27, 2026 15:41
@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Oct 27, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc
SunMarc added this pull request to the merge queue Oct 27, 2026
Merged via the queue into huggingface:main with commit 455c9a0 Oct 27, 2026
127 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SunMarc SunMarc SunMarc approved these changes
@ydshieh ydshieh Awaiting requested review from ydshieh

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kylesayrs @HuggingFaceDocBuilderDev @SunMarc