Repository navigation
Add xcodec2 model #44178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Add xcodec2 model #44178
Changes from 130 commits
Commits
Show all changes
137 commits
Select commit
Hold shift + click to select a range
277a96f
Add xcodec model
349feae
code formatting
e5f1da8
typo xcodec2 name
Deep-unlearning fc0907c
add xcodec2 in init file
Deep-unlearning ea0acbf
fix import
Deep-unlearning 8542db7
fix weight_norm init
Deep-unlearning e98d981
remove unused import
Deep-unlearning d1cd3ac
add convert file
Deep-unlearning 74fa506
add ModelOutput class
Deep-unlearning 02f5c94
nit
Deep-unlearning dd0a17c
fix device issue
3786203
fix forward
93dbfad
nit
d4d8c6a
doc draft
c40912e
draft test
17eb48c
match tensor with the orignal implementation
3760438
Add doc file for xcodec2
a2faa55
finish model doc for xcodec2
31319fb
update doc
Deep-unlearning 8d9f8df
working xcodec2
Deep-unlearning e5a1838
add test file for xcodec2
Deep-unlearning 473f95a
nit
Deep-unlearning dd8aace
xcodec2 use EncodecFeatureExtractor
Deep-unlearning f6cf875
Merge branch 'main' into add-xcodec2
ebezzam 244bdb6
Standardize with Xcodec.
ebezzam a84a69f
Merge branch 'main' into add-xcodec2
ebezzam 9d743e8
Merge branch 'main' into add-xcodec2
ebezzam 2e23505
Address some PR comments and standardize.
ebezzam 0316080
Remove Sequential.
ebezzam fcbeab7
Remove weight norm from model definition.
ebezzam 3c50dd2
Remove padding.
ebezzam bfe535b
Better use of modular and better init.
ebezzam f287f6a
Style and format checks.
ebezzam 88cc8a7
Address some modeling tests.
ebezzam 8eddd59
Better use of modular for Attention and cleanup.
ebezzam dda588b
Remove asserts, expose params in config.
ebezzam 1218679
Clean up internal and better docstrings.
ebezzam 5378c81
Fix example.
ebezzam 1005594
Make clear how self.causal is used
ebezzam ee09e64
Add padding consistent to original.
ebezzam 857562f
Clean up and matching integration test.
ebezzam a92c67c
Update docs and clean up.
ebezzam de3a7f8
Correct doc location among audio models.
ebezzam 767208b
Remove flash tests.
ebezzam 48a69c2
Skip flash attention tests.
ebezzam 2dfb96e
Switch to processor.
ebezzam 1cb9e89
New feature extractor.
ebezzam 892d8e8
Update modular.
ebezzam edf0738
Add feature extractor.
ebezzam 185afea
Clean feature extractor and expose some parameters.
ebezzam 755aaec
Clean up feature extractor and add torch support (wip).
ebezzam 2a2d037
Clean up seamless feature extractor.
ebezzam af636f9
Modular cleanup.
ebezzam fcb0ee5
Remove processor.
ebezzam e10b04a
Update docs.
ebezzam bc50545
Fix copy paths.
ebezzam 5493319
Fix modeling tests for audio_spectrogram input.
ebezzam 2908abd
Fix torch support with new spectrogram torch utility.
ebezzam cf9fd28
Repo consistency.
ebezzam d97b48c
Feature extraction tests.
ebezzam 3c0a5c7
Merge branch 'main' into add-xcodec2
ebezzam 4d78a95
Make style happy
ebezzam bd9f37f
Remove unprotected import.
ebezzam 3d365e5
Another unprotected import.
ebezzam 6b812a0
Remove more unprotected torches.
ebezzam 2bfc30a
zero_mean_unit_var_norm needed for a test
ebezzam 1911438
Modify Vocos component to be able to use modular later.
ebezzam dd3f45f
Update modular
ebezzam 8cfce62
Make style happy.
ebezzam a76717a
Merge branch 'main' into add-xcodec2
ebezzam 3650d87
Regenerate modular, update rope config.
ebezzam a4cca27
Update init weights
ebezzam e93c846
Add to init weights
ebezzam 86ee458
FIxed _init_weights
ebezzam 1a8b461
Integration passing!
ebezzam a2dbfdb
Cleanup
ebezzam 4129a27
Better use of modular, and cleanup.
ebezzam ab62936
Merge branch 'main' into add-xcodec2
ebezzam 4ffff44
Update config after mering with main, and other nits.
ebezzam b5ec79a
Merge branch 'main' into add-xcodec2
ebezzam 642b56c
Address tests and other nits.
ebezzam 1fbe78d
Nits, better wav2vec2 bert init
ebezzam 8570ded
CLean up feature extraction and other nits
ebezzam 15e9349
doc nits
ebezzam 7bd81c4
Nit
ebezzam a97cc2c
Simplify feature extraction.
ebezzam 29ce7fd
Torch compiile compatibility.
ebezzam 02e92ac
Rename output entries, semantic encoder. Move weight norm utilities. …
ebezzam 811aa2a
Add batch test and spectrogram padding for equivalence.
ebezzam 672e1d2
Refactoring.
ebezzam d221fbf
Remove defensive trimming.
ebezzam c2ab39c
Merge branch 'main' into add-xcodec2
ebezzam 4c6f001
Make style happy.
ebezzam 6a01e49
Apply suggestion from @vasqu
ebezzam dad49b0
Fix feature extraction tests, remove integration since original doesn…
ebezzam 72f19cf
Update date.
ebezzam 49d3671
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam 7aa53eb
Address docs comments.
ebezzam 63a86b8
Apply suggestion from @vasqu
ebezzam 4755347
Apply suggestion from @vasqu
ebezzam 17a4b2b
Address various comments.
ebezzam 7fd8f9e
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam f2ee118
Add import for RopeParameters
ebezzam 0ba5a74
Address feature extractor comments.
ebezzam 9e3a38d
Adding note on spectrogram computation.
ebezzam 1132c46
Address smaller modeling comments.
ebezzam 0473f59
Address various modeling comments.
ebezzam f5d2968
Merge branch 'main' into add-xcodec2
ebezzam fc2abde
Style fixes after merge.
ebezzam eacfd3b
better api for xcodec forward
ebezzam 2ba74c2
Better module naming in Qwen2.5 Omni
ebezzam b88a748
Address various comments.
ebezzam c7e3247
Add comments, add audio utils, address nits.
ebezzam 4126e85
Merge branch 'main' into add-xcodec2
ebezzam b658be7
Keep current audio input names, various nits.
ebezzam 070af74
Add casting for cpu
ebezzam ffddbcc
Merge branch 'main' into add-xcodec2
ebezzam 7a99db7
Move utils for testing audio.
ebezzam 2a02a01
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam 4320269
Merge branch 'main' into add-xcodec2
ebezzam b8df2c7
Merge branch 'main' into add-xcodec2
ebezzam 65f73f1
Update checkpoint name.
ebezzam e92c4c4
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam f230672
Merge branch 'main' into add-xcodec2
ebezzam 8c9b8d9
FIx for torch compile
ebezzam 7d91a28
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam dc3378d
Merge branch 'main' into add-xcodec2
ebezzam 0142779
Set test config for offload to pass.
ebezzam 512c681
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam a31ed1c
Increase allowed num layers.
ebezzam 78c3ed9
Config that passes cpu offload.
ebezzam 6d9c51c
test config for disk/cpu offload
ebezzam a5c07db
test nits
ebezzam 2fe0150
Merge branch 'main' into add-xcodec2
ebezzam 12e6464
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam 975813f
update date
ebezzam 6242646
Merge branch 'main' into add-xcodec2
ebezzam File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| <!--Copyright 2026 The HuggingFace Team. All rights reserved. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations under the License. | ||
|
|
||
| ⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be | ||
| rendered properly in your Markdown viewer. | ||
|
|
||
| --> | ||
| *This model was published in HF papers on 2025-02-06 and contributed to Hugging Face Transformers on 2026-06-23.* | ||
|
|
||
| # X-Codec2 | ||
|
|
||
| <div class="flex flex-wrap space-x-1"> | ||
| <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white"> | ||
| </div> | ||
|
|
||
| ## Overview | ||
|
|
||
| The X-Codec2 model was proposed in [Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis](https://huggingface.co/papers/2502.04128). | ||
|
|
||
| X-Codec2 is a neural audio codec designed to improve speech synthesis and general audio generation for large language model (LLM) pipelines. It extends the original X-Codec by refining how semantic and acoustic information is integrated and tokenized, enabling efficient and high-fidelity audio representation. | ||
|
|
||
| About its architecture: | ||
| - **Unified Semantic-Acoustic Tokenization**: X-Codec2 fuses outputs from a semantic encoder (e.g., Wav2Vec2-BERT) and an acoustic encoder into a single embedding, capturing both high-level meaning (e.g., text content, emotion) and low-level audio details (e.g., timbre). | ||
| - **Single-Stage Feature Scalar Quantization (FSQ)**: Unlike the multi-layer residual VQ in most approaches (e.g., [DAC](./dac), [EnCodec](./encodec), [X-Codec](./xcodec), [Mimi](./mimi.md)), X-Codec2 uses a single-layer of Feature Scalar Quantization (FSQ) for stability and compatibility with causal, autoregressive LLMs. | ||
| - **Transformer-Friendly Design**: The 1D token structure of X-Codec2 naturally aligns with the autoregressive modeling in LLMs like LLaMA, improving training efficiency and downstream compatibility. | ||
|
|
||
| A model checkpoint is available at [HKUSTAudio/xcodec2-hf](https://huggingface.co/HKUSTAudio/xcodec2-hf). | ||
|
|
||
| This model was contributed by [Eric Bezzam](https://huggingface.co/bezzam) and [Steven Zheng](https://huggingface.co/Steveeeeeeen). | ||
| The original modeling code can be found [here](https://huggingface.co/HKUSTAudio/xcodec2/blob/main/modeling_xcodec2.py), while their training code is [here](https://github.com/zhenye234/X-Codec-2.0). | ||
|
|
||
| ## Usage example | ||
|
|
||
| Here is a quick example of how to encode and decode an audio using this model: | ||
|
|
||
| ```python | ||
| from datasets import Audio, load_dataset | ||
| from transformers import AutoFeatureExtractor, AutoModel | ||
|
|
||
| model_id = "HKUSTAudio/xcodec2-hf" | ||
| model = AutoModel.from_pretrained(model_id, device_map="auto") | ||
| feature_extractor = AutoFeatureExtractor.from_pretrained(model_id) | ||
|
|
||
| dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") | ||
| dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) | ||
| audio = dataset[0]["audio"]["array"] | ||
| inputs = feature_extractor(audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to( | ||
| model.device, model.dtype | ||
| ) | ||
| print("Input waveform shape:", inputs["input_values"].shape) | ||
| # Input waveform shape: torch.Size([1, 1, 93760]) | ||
|
|
||
| # encoder and decoder | ||
| audio_codes = model.encode(**inputs).audio_codes | ||
| print("Audio codes shape:", audio_codes.shape) | ||
| # Audio codes shape: torch.Size([1, 1, 293]) | ||
| audio_values = model.decode(audio_codes).audio_values | ||
| print("Audio values shape:", audio_values.shape) | ||
| # Audio values shape: torch.Size([1, 1, 93760]) | ||
|
|
||
| # Equivalently, you can do encoding and decoding in one step | ||
| model_output = model(**inputs) | ||
| audio_codes = model_output.audio_codes | ||
| audio_values = model_output.audio_values | ||
| ``` | ||
|
|
||
| ### Batch processing | ||
|
|
||
| This implementation also supports batched input, unlike the original [release](https://huggingface.co/HKUSTAudio/xcodec2)! | ||
|
|
||
| ```python | ||
| from datasets import Audio, load_dataset | ||
| from transformers import AutoFeatureExtractor, AutoModel | ||
|
|
||
| batch_size = 2 | ||
| model_id = "HKUSTAudio/xcodec2-hf" | ||
| model = AutoModel.from_pretrained(model_id, device_map="auto") | ||
| feature_extractor = AutoFeatureExtractor.from_pretrained(model_id) | ||
|
|
||
| dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") | ||
| dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) | ||
| audios = [dataset[i]["audio"]["array"] for i in range(batch_size)] | ||
| inputs = feature_extractor(audio=audios, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to( | ||
| model.device, model.dtype | ||
| ) | ||
| print("Input waveform shape:", inputs["input_values"].shape) | ||
| # Input waveform shape: torch.Size([2, 1, 93760]) | ||
|
|
||
| # encoder and decoder | ||
| encoder_output = model.encode(**inputs) | ||
| audio_codes = encoder_output.audio_codes | ||
| print("Audio codes shape:", audio_codes.shape) | ||
| # Audio codes shape: torch.Size([2, 1, 293]) | ||
| audio_values = model.decode(audio_codes).audio_values | ||
| print("Audio values shape:", audio_values.shape) | ||
| # Audio values shape: torch.Size([2, 1, 93760]) | ||
|
|
||
| # Equivalently, you can do encoding and decoding in one step | ||
| model_output = model(**inputs) | ||
| audio_codes = model_output.audio_codes | ||
| audio_values = model_output.audio_values | ||
| ``` | ||
|
|
||
| ### Speed-up with `torch.compile` | ||
|
vasqu marked this conversation as resolved.
Resolved
|
||
|
|
||
| You can speed up inference with [`torch.compile`](https://pytorch.org/docs/stable/generated/torch.compile.html). The first few calls will be slower due to compilation overhead, but subsequent calls will be faster. | ||
|
|
||
| On an A100, we observed a speed-up of ~1.35 for a batch size of 4 ([script](https://gist.github.com/ebezzam/3b79481b5d48d8e35c4ecc582aee0cb3#file-benchmark_torch_compile-py)). | ||
|
|
||
| ```python | ||
| import torch | ||
| from datasets import Audio, load_dataset | ||
| from transformers import AutoFeatureExtractor, AutoModel | ||
|
|
||
| batch_size = 4 | ||
| model_id = "HKUSTAudio/xcodec2-hf" | ||
| model = AutoModel.from_pretrained(model_id, device_map="auto") | ||
| feature_extractor = AutoFeatureExtractor.from_pretrained(model_id) | ||
|
|
||
| dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") | ||
| dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) | ||
| audios = [dataset[i]["audio"]["array"] for i in range(batch_size)] | ||
| inputs = feature_extractor( | ||
| audio=audios, sampling_rate=feature_extractor.sampling_rate, padding=True, return_tensors="pt" | ||
| ).to(model.device, model.dtype) | ||
|
|
||
| compiled_model = torch.compile(model, fullgraph=True) | ||
|
|
||
| # Warmup (includes compilation on first call) | ||
| for _ in range(10): | ||
| with torch.inference_mode(): | ||
| _ = compiled_model(**inputs) | ||
|
|
||
| with torch.inference_mode(): | ||
| output = compiled_model(**inputs) | ||
| print("Audio values shape:", output.audio_values.shape) | ||
| ``` | ||
|
|
||
| ## Xcodec2Config | ||
|
|
||
| [[autodoc]] Xcodec2Config | ||
|
|
||
| ## Xcodec2FeatureExtractor | ||
|
|
||
| [[autodoc]] Xcodec2FeatureExtractor | ||
| - __call__ | ||
|
|
||
| ## Xcodec2Model | ||
|
|
||
| [[autodoc]] Xcodec2Model | ||
| - decode | ||
| - encode | ||
| - forward | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
ebezzam marked this conversation as resolved.
Resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.